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ABSTRACT 


The  Training  &  Doctrine  Command  (TRADOC)  Analysis  Center  (TRAC)  - 
Monterey,  California  re-engineered  the  Janus  simulation  as  a  technology  demonstration. 
The  completed  simulation,  HLA  Warrior,  applied  modem  technologies  including  an 
object-oriented  design  and  state-of-art  user  interfaces.  The  project  also  re-wrote  Janus 
source  code  in  C++.  The  purpose  of  this  thesis  was  to  assess  HLA  Warrior’s  fidelity, 
defined  as  its  ability  to  replicate  Janus  results,  by  conducting  a  statistical  comparison  of 
Janus  and  HLA  Warrior.  Given  that  Janus  has  high  “face-validity,”  Janus  results  acted  as 
the  baseline  from  which  HLA  Warrior  results  were  compared.  The  comparison  involved 
executing  identical  scenarios  in  Janus  and  HLA  Warrior,  gathering  results,  and 
conducting  a  rigorous  statistical  comparison  of  Janus  and  HLA  Warrior  results. 

Statistical  tests  included  the  paired  r-test  and  non-parametric  Wilcoxon  Signed  Ranks 
Test. 

Results  from  the  tests  showed  differences  between  Janus  and  HLA  Warrior. 
Investigation  into  the  causes  of  the  differences  found  two  source  code  errors  in  HLA 
Warrior.  Re-evaluation  of  HLA  Warrior  following  correction  of  the  errors  resulted  in  a 
reduction  in  magnitude  of  the  differences.  Probable  causes  due  to  algorithm 
implementation  differences  were  also  identified.  While  differences  exist,  HI  .A  Warrior 
appears  to  have  face-validity  and  generally  produces  outcomes  similar  to  Janus. 
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EXECUTIVE  SUMMARY 


The  United  States  Army’s  Training  and  Doctrine  Command  Analysis  Center- 
Monterey  (TRAC-Monterey)  recently  re-engineered  the  Janus  simulation  as  a  technology 
demonstration.  Technologies  included  an  innovative  architecture,  state-of-art  user 
interfaces,  and  built-in  High-Level  Architecture  (HLA)  tools.  The  project  ported  Janus  to 
a  personal  computer  (PC)  running  the  Windows  NT  (WinNT)  operating  system  and 
rewrote  Janus  source  code  in  C++  while  maintaining  the  integrity  of  the  Janus 
algorithms.  The  resulting  simulation  is  known  as  HLA  Warrior. 

The  purpose  of  this  thesis  was  to  assess  HLA  Warrior’s  fidelity,  defined  as  the 
simulation’s  ability  to  replicate  Janus  results,  by  conducting  a  statistical  comparison  of 
Janus  and  HLA  Warrior  scenarios.  The  research  determined  whether  results  from  Janus 
and  HLA  Warrior  scenarios  were  statistically  similar.  Since  Janus  has  high  “face- 
validity,”  it  acted  as  the  baseline  to  compare  HLA  Warrior’s  results.  A  model  with  high 
face  validity  is  defined  as  a  model  that,  on  the  surface,  seems  reasonable  to  people  who 
are  knowledgeable  about  the  system  under  study  [Ref.  l:p.  308]. 

Janus  is  a  high-resolution,  ground  combat  simulation  focusing  on  maneuver  and 
artillery  units.  Janus  represents  entities  down  to  individual  systems  or  soldiers.  Janus 
stochastically  adjudicates  all  detections  and  engagements  between  individual  systems. 
Janus  contains  a  post-processor  that  permits  the  analyst  to  gather  comprehensive  statistics 
on  the  simulation  run,  including  reports  on  the  number  of  rounds  fired,  detection  ranges, 
kill  ranges.  Force  Exchange  Ratio  (FER),  and  Loss  Exchange  Ratios  (LER). 

To  assess  whether  HLA  Warrior  can  replicate  Janus  scenario  results,  the  research 
analyzed  measures  of  performance  (MOPs)  that  quantified  the  engagement  process.  The 
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fundamental  steps  of  an  engagement  sequence  include  detecting  then  shooting  an  entity. 
Selected  MOPs  included  detection  range,  kill  range,  and  number  of  rounds  fired.  The 
research  also  analyzed  force  exchange  ratio  (PER)  as  a  quantitative  measure  of  overall 
battle  outcome. 

Two  scenarios  were  developed  to  analyze  results  from  varying  weapons  systems: 
a  mechanized  scenario  and  a  light  infantry  scenario.  Each  scenario  was  then  executed  in 
two  environments.  Fort  Hunter-Liggett,  California  and  Southwest  Asia.  These  distinctly 
different  environments  support  the  analysis  as  to  whether  HLA  Warrior  successfully 
models  the  engagement  process  under  varying  line-of-sight  conditions. 

Each  of  the  two  scenarios  (mechanized,  light)  was  executed  in  two  environments 
(Southwest  Asia,  Fort  Hunter-Liggett)  and  run  in  two  different  modes  (Janus,  HLA 
Warrior).  The  two  scenarios,  two  environments,  and  two  modes  resulted  in  a  total  of 
eight  combinations  for  the  experiment.  Each  of  the  combinations  was  executed  10  times. 
In  total,  80  runs  were  performed  to  provide  a  reasonable  data  sample  to  analyze  each 
MOP. 

The  method  of  analysis  was  to  compare  the  MOPs  resulting  from  a  specific 
number  of  Janus  runs  to  the  MOPs  resulting  from  the  same  number  of  HLA  Warrior  runs. 
Since  Janus  produces  exactly  the  same  results  when  the  same  random  number  seed  is 
used  (likewise  for  HLA  Warrior),  all  Janus  runs  were  executed  using  different,  randomly 
selected  seeds.  The  same  random  number  seeds  were  then  used  in  the  corresponding 
HLA  Warrior  runs.  The  combination  of  the  same  scenario,  same  environment,  and 
same  random  number  seed  executed  in  opposing  modes  (Janus  and  HLA  Warrior) 
support  paired  output  from  the  two  simulations. 
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After  completing  all  runs  and  gathering  required  data,  MOPs  were  analyzed  using 
the  Wilcoxon  Signed  Ranks  Test.  Then  those  MOPs  that  satisfied  the  paired  f-test’s 
normality  assumption  as  determined  by  observation  of  normal  probability  plots  were 
analyzed  using  the  more  powerful  paired  r-test. 

The  results  of  the  analysis  showed  a  disparity  between  HLA  Warrior  and  Janus. 
Further  investigation  into  the  causes  of  the  disparity  led  to  two  possible  explanations. 
First,  two  errors  in  the  HLA  Warrior  source  code  were  found.  Re-evaluation  of  HLA 
Warrior  following  correction  of  the  source  code  errors  resulted  in  a  reduction  in  the 
magnitude  of  the  differences  between  Janus  and  HLA  Warrior.  A  second  possible  cause 
of  the  differences  between  the  simulations  was  a  difference  in  the  algorithm 
implementation  methodology  between  Janus  and  HLA  Warrior.  While  the  integrity  of 
the  Janus  algorithms  were  maintained  in  HLA  Warrior,  developers  implemented 
algorithms  differently  in  HLA  Warrior  to  improve  efficiency.  The  implementation 
differences  could  lead  to  a  disparity  in  the  results  of  the  investigated  MOPs. 

While  the  two  simulations  are  different  with  regards  to  selected  MOPs,  HI  .A 
Warrior  generally  produces  similar  outcomes  to  Janus.  In  general,  the  flow  of  the  battles 
executed  in  HLA  Warrior  progressed  as  a  subject  matter  expert  might  expect,  lending  a 
degree  of  face-validity  to  HLA  Warrior. 
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I.  INTRODUCTION 


The  United  States  Army’s  Training  and  Doctrine  Command  (TRADOC)  Analysis 
Center-Monterey,  California  (TRAC-Monterey)  recently  re-engineered  the  Janus 
simulation  with  modem  technologies,  including  an  innovative  architecture,  state-of-art 
user  interfaces,  and  built-in  High-Level  Architecture  (HLA)  tools.  The  project  ported 
Janus  to  a  personal  computer  (PC)  mnning  the  Windows  NT  (WinNT)  operating  system 
and  rewrote  Janus  source  code  in  C++.  The  resulting  simulation  is  known  as  HLA 
Warrior.  HLA  Warrior’s  primary  objective  was  to  serve  as  a  technology  demonstration, 
providing  lessons  learned  and  reusable  products  to  developers  of  future  simulations. 

The  purpose  of  this  thesis  was  to  assess  HLA  Warrior’s  fidelity,  defined  as  the 
simulation’s  ability  to  replicate  Janus  results,  by  conducting  a  statistical  comparison  of 
Janus  and  HLA  Warrior  scenarios.  The  research  determined  whether  results  from  Janus 
and  HLA  Warrior  scenarios  were  statistically  similar.  Since  Janus  has  high  “face- 
validity,”  it  acted  as  a  baseline  to  compare  HLA  Warrior’s  results.  A  model  with  high 
face  validity  is  defined  as  a  model  that,  on  the  surface,  seems  reasonable  to  people  who 
are  knowledgeable  about  the  system  under  study  [Ref.  l:p.  308].  Subject  matter  experts 
(infantry  officers,  armor  officers,  and  so  on)  have  examined  the  execution  of  various 
Janus  scenarios  and  determined  that  the  way  in  which  the  battles  progressed  and  the 
outcomes  of  the  battles  were  plausible  and  likely  replicated  real-world  phenomena. 

In  order  to  test  for  similarity  between  Janus  and  HLA  Warrior,  measures  of 
performance  (MOPs)  were  developed.  Where  the  simulation  results  were  statistically 
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similar,  the  findings  aided  in  the  validation  of  HLA  Warrior.  The  thesis  also  identified 
factors  that  may  account  for  any  significant  differences  between  the  two  simulations. 

A.  JANUS 

Lawrence  Livermore  National  Laboratory  originally  developed  Janus  in  the 
1970s,  with  the  current  version  developed  by  TRAC-White  Sands  Missile  Range,  New 
Mexico.  Most  Janus  systems  use  a  UNIX  operating  system  and  execute  on  Hewlett- 
Packard  workstations. 

Janus  is  a  ground  combat  simulation  primarily  focusing  on  maneuver  and  artillery 
units.  Janus  also  models  engineer  support,  minefield  employment  and  breaching,  rotary 
and  fixed-wing  aircraft,  chemical  environments,  limited  weather  effects,  and  day  and 
night  visibility.  Janus  is  a  high-resolution  simulation,  representing  entities  down  to 
individual  systems  or  soldiers.  The  user  may  aggregate  entities  when  appropriate. 

[Ref  2:p.  21] 

Analysts  use  Janus  to  conduct  in-depth  studies  of  tactics,  techniques,  and 
procedures.  Analysts  can  also  use  Janus  to  evaluate  new  weapon  systems  in  various 
environments,  weather  conditions,  and  mission  profiles.  Janus  is  also  used  to  train 
brigade  and  battalion  staffs. 

Janus  is  interactive  in  that  interplay  between  operators  during  simulated  combat  is 
possible.  Users  may  influence  scenario  results  by  altering  movement  routes,  planning 
and  firing  artillery  missions,  and  mounting/dismounting  entities  on  vehicles.  User 
interaction  is  not  necessary  for  pre-planned  scenarios.  For  purposes  of  this  thesis,  all 
scenarios  were  pre-planned  and  included  no  user  interaction  during  execution. 
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Janus  stochastically  adjudicates  all  detections  and  engagements  between 
individual  systems.  Prior  to  the  Janus  run,  the  user  may  specify  a  random  number  seed 
up  to  seven  digits,  allow  the  system  to  randomly  specify  a  seed,  or  use  the  default  random 
number  seed  hard-coded  into  the  program  [Ref.  2:p.  42].  When  the  same  random  number 
seed  is  used  in  the  same  scenario  without  human  interaction,  the  exact  same  results  occur. 

Janus  possesses  a  robust  database  that  permits  the  user  to  define  a  weapon  system 
extensively  and  capture  detailed  factors  required  for  scenario  development.  Individual 
fighting  systems  have  distinct  properties:  dimensions,  weight,  carrying  capacity,  and 
speed.  Users  can  also  modify  weapons  system  parameters  such  as  range,  ordnance  type, 
and  ammunition  basic  load  (supply  of  ammunition  each  entity  carries  into  a  battle) 

[Ref.  2:p.  4]. 

Janus’  post-processor  permits  analysts  to  gather  comprehensive  statistics  on  the 
simulation  run,  including  reports  on  the  detection  ranges,  kill  ranges,  chemical  casualties. 
Force  Exchange  Ratio  (PER),  number  of  rounds  fired,  and  Loss  Exchange  Ratios  (LER). 

B.  HLA  WARRIOR 

TRAC-Monterey  re-engineered  the  Janus  simulation  with  several  modem 
technologies,  including  an  innovative  architecture,  windows-like  user  interfaces,  and 
HLA  tools.  The  project  ported  Janus  to  a  PC  running  the  WinNT  operating  system  and 
rewrote  Janus  source  code  in  C-H-  while  maintaining  the  integrity  of  the  Janus 
algorithms.  HLA  Warrior  implements  new  graphical  user  interfaces  (GUIs)  using  Vision 
XXI  management  tools  that  comply  with  Distributed  Interactive  Simulation  (DIS)  and 
HLA  requirements.  The  U.S.  Army  National  Simulation  Center  (NSC)  will  also  expand 
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HLA  Warrior  capability  by  integrating  Operations  Other  Than  War  (OOTW)  algorithms 
from  the  Spectrum  simulation.  [Ref.  3;p.  1]. 

HLA  Warrior  possesses  all  of  the  modeling  and  anal)^ical  capabilities  of  Janus. 
HLA  Warrior  also  has  the  capability  to  read  and  execute  scenarios  developed  in  Janus 
including  entity  locations  and  movement  routes.  This  function  proved  useful  in  the 
design  of  the  experiment  because  once  a  scenario  is  developed  in  Janus,  its  initial  settings 
can  be  duplicated  in  HLA  Warrior  prior  to  the  run. 
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II.  PROBLEM  DESCRIPTION 


The  purpose  of  this  thesis  was  to  assess  HLA  Warrior’s  fidelity,  defined  as  the 
simulation’s  ability  to  replicate  Janus  results,  by  conducting  a  statistical  comparison  of 
Janus  and  HLA  Warrior  scenarios.  To  answer  this  question,  the  thesis  tested  HLA 
Warrior’s  ability  to  accurately  replicate  specific  combat  functions  under  varying 
conditions.  Of  primary  concern  is  the  ability  of  HLA  Warrior  to  replicate  the  basic 
engagement  process,  detecting  and  killing  a  target. 

A  primary  element  of  detecting  and  subsequent  killing  of  a  target  in  ground 
combat  is  line  of  sight.  The  target  must  be  in  the  line  of  sight  of  the  shooter  in  order  to  be 
detected.  Further,  the  target  must  remain  within  line  of  sight  of  a  direct  fire  weapon 
throughout  the  engagement  process  to  be  killed.  Therefore,  HLA  Warrior  must 
accurately  replicate  Janus’  line-of-sight  calculations. 

Provided  the  line-of-sight  calculations  are  accurate,  HLA  Warrior  must  replicate 
Janus’  detection  and  kill  algorithms  accurately  in  order  for  HLA  Warrior  to  be  similar  to 
Janus.  Both  Janus  and  HLA  Warrior  use  a  stochastic  process  to  adjudicate  detections  and 
kills.  This  implies  that  both  the  random  number  generators  and  the  algorithms  that 
assess  the  probabilities  of  hit  and  kill  in  the  two  simulations  need  to  be  similar. 

This  thesis  therefore  provided  a  method  to  assess  HLA  Warrior’s  line-of-sight 
algorithms,  random  number  generators,  detection  algorithms,  and  kill  algorithms  for 
comparison  to  Janus. 
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The  following  chapter  describes  the  design  of  the  experiment  to  determine 
whether  Janus  and  HLA  Warrior  are  statistically  similar  and  address  the  experimental 
issues  raised  above. 
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III.  DESIGN  OF  EXPERIMENT 


The  goal  of  the  experiment  was  to  provide  a  means  to  obtain  data  from  identical 
scenarios  executed  in  both  Janus  and  HLA  Warrior  for  the  purpose  of  analysis.  Design 
issues  included  selecting  appropriate  and  sufficient  data  to  be  analyzed,  determining  the 
nature  of  the  scenarios,  and  conducting  the  experiment. 

A.  MEASURES  OF  PERFORMANCE 

The  first  issue  in  the  design  of  the  experiment  was  to  identify  and  select  data  to  be 
analyzed.  The  Measures  of  Performance  (MOPs)  were  limited  to  information  available 
from  the  simulations’  post-processors.  The  MOPs  also  had  to  support  the  analysis  of  the 
issues  raised  in  the  Problem  Description.  To  assess  whether  HLA  Warrior  can  replicate 
Janus  scenario  results,  the  research  analyzed  MOPs  that  quantified  the  engagement 
process.  The  fundamental  steps  of  an  engagement  sequence  include  detecting  then 
shooting  an  entity.  Selected  MOPs  included  detection  range,  kill  range,  and  number  of 
rounds  fired.  The  research  also  analyzed  force  exchange  ratio  (PER)  as  a  quantitative 
measure  of  overall  battle  outcome. 

1.  Detection  Range 

Detecting  a  target  is  the  first  basic  element  of  a  battle.  Before  a  combat  system 
can  engage  and  subsequently  kill  a  target,  the  target  must  first  be  detected.  Comparing 
the  detection  ranges  produced  by  Janus  and  HLA  Warrior  is  a  quantitative  method  of 
assessing  the  line-of-sight  calculations  as  well  as  the  detection  algorithms  in  HLA 
Warrior. 
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2. 


Kill  Range 


The  second  basic  element  in  a  battle  is  killing  a  target  once  detected.  Analyzing 
the  kill  ranges  of  Janus  and  HLA  Warrior  provides  evaluation  of  the  line-of-sight 
algorithms  and  a  quantitative  method  for  evaluating  HLA  Warrior’s  engagement  process. 

3.  Force  Exchange  Ratio 

Generally,  a  combat  simulation’s  overall  objective  is  to  accurately  determine  the 
victor  in  a  given  engagement.  The  force  exchange  ratio,  defined  by  the  equation  below, 

(J^^dlosses  /  Blu€i(,sses)  ^  initial  /  BluCiniiial) 

is  a  quantitative  measure  of  the  scenario  outcome.  In  words,  the  FER  is  the  quantity  of 
the  loss  exchange  ratio  divided  by  the  initial  force  ratio.  The  FER  is  a  more  accurate 
indicator  of  the  outcome  than  the  loss  exchange  ratio  (total  red  side  losses  divided  by 
total  blue  side  losses)  because  it  takes  initial  force  levels  into  account  and  standardizes 
losses.  That  is,  if  two  opposing  forces  lose  an  equal  number  of  systems,  losses  to  the 
smaller  force  are  more  damaging  than  losses  to  the  larger  force. 

4.  Rounds  Fired 

Not  all  shots  result  in  kills.  Therefore,  analysis  of  the  total  number  of  rounds 
fired  in  both  models  is  useful  to  assess  the  engagement  process.  Comparing  the  total 
number  of  rounds  fired  in  both  simulations  can  also  support  analysis  of  Janus’  kill 
algorithms  and  probability  of  kill  processes  being  replicated  in  HLA  Warrior. 

B.  SCENARIOS 

The  scenarios  were  developed  based  on  the  requirements  posed  in  the  Problem 
Description.  It  is  important  to  analyze  results  from  varied  scenarios  to  model  varying 
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weapons  systems  and  their  capabilities.  Therefore  the  experiment  included  scenarios 
using  dissimilar  systems,  an  armored/mechanized  scenario  and  a  light  infantry  scenario. 

1.  Mechanized  Forces 

The  first  scenario  was  an  armored/mechanized  force  scenario.  The  scenario 
consisted  of  a  tank  company  reinforced  with  armored  personnel  carriers  defending  and  a 
similarly  equipped  battalion  attacking.  Both  sides  were  armed  with  Ml  A1  Abrams  main 
battle  tanks  and  M2  Bradley  armored  personnel  carriers  (APCs).  The  defenders  had  14 
tanks  and  four  APCs.  The  attacking  force  consisted  of  39  tanks  and  eight  APCs.  The 
initial  force  ratio  of  attacker  to  defender  was  approximately  3: 1  for  armored  vehicles. 
Additionally,  both  sides  were  equipped  with  two  AH-64  Apache  helicopters  as  well  as  air 
defense  systems. 

2.  Light  Forces 

The  second  scenario  was  a  light  infantry  scenario  consisting  of  a  rifle  company 
defending  and  a  rifle  battalion  attacking.  The  defending  company  consisted  of  103 
entities  comprised  of  four  system  types  including  riflemen,  light  machine  guns,  machine 
guns,  and  light  anti-tank  weapons  (LAW).  The  attacking  battalion  was  similarly 
equipped  with  a  total  of  285  entities,  resulting  in  an  initial  force  ratio  of  approximately 
3:1. 

C.  ENVIRONMENTS 

In  order  to  assess  HLA  Warrior’s  detection  and  kill  algorithms,  the  selected 
scenarios  were  executed  in  contrasting  environments.  A  hilly,  wooded  environment  and 
a  flat,  desert  environment  were  selected  for  the  experiment.  These  distinct  environments 
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support  the  analysis  as  to  whether  HLA  Warrior  accurately  models  the  detection  and  kill 
algorithms  under  varying  line-of-sight  conditions. 

1.  Fort  Hunter-Liggett  (HL),  California 

The  Fort  Hunter-Liggett  terrain  file  provided  a  hilly,  wooded  environment.  The 
hills  and  trees  test  the  line-of-sight  calculations  and,  consequently,  the  detection  and  kill 
algorithms  in  a  constrained  visual  environment.  The  hills,  when  combined  with  the 
effects  of  the  trees,  serve  to  limit  line  of  sight  and  adversely  effect  the  detection  and  kill 
process. 

2.  Southwest  Asia  (SWA) 

The  Southwest  Asia  terrain  file  provided  a  relatively  flat,  featureless,  desert 
environment.  The  terrain  is  less  restrictive  to  line-of-sight  than  Hunter-Liggett  terrain. 
Therefore  the  detection  and  kill  ranges  should  approach  maximum  effective  range  of 
individual  sensors  and  weapon  systems. 

D.  MODELS 

After  determining  the  scenarios  and  environments,  four  separate  models  were 
developed:  one  mechanized  and  one  light  scenario  in  Southwest  Asia  and  one 
mechanized  and  one  light  infantry  scenario  at  Fort  Hunter-Liggett.  Modification  of  the 
Janus  weapons  system  database  ensured  that  each  weapon  system  on  one  side  of  the 
battle  could  engage  each  type  of  system  on  the  opposing  side. 

In  each  scenario,  the  blue  force  (the  defending  force)  was  positioned  on  easily 
defendable  terrain.  The  fields  of  view  vary  from  entity  to  entity  to  adequately  test  the 
line-of-sight  calculations.  All  blue  force  ground  entities  (tanks,  APCs,  infantrymen,  etc.) 
remained  stationary  throughout  the  battle.  The  blue  AH-64s  in  the  mechanized  scenarios 
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traveled  along  pre-planned  routes.  Both  AH-64s  began  movement  as  the  simulation 
commenced. 

Each  entity  in  the  red  force  (the  attacking  force)  traveled  along  a  pre-planned 
movement  route  toward  an  objective  occupied  by  the  blue  force.  Additionally,  the  red 
force  AH-64s  in  the  mechanized  scenarios  traveled  along  a  route  that  took  them  over  the 
blue  force.  All  red  force  movement  commenced  as  the  simulation  began. 

Although  one  of  the  purposes  behind  the  development  of  HLA  Warrior  was  to 
ease  the  scenario  building  process,  no  scenarios  were  constructed  in  HLA  Warrior. 
Rather,  HLA  Warrior’s  ability  to  read  and  convert  Janus  scenario  files  was  utilized.  This 
served  to  ensure  that  the  four  models  executed  in  HLA  Warrior  had  starting  positions, 
routes,  and  pre-planned  missions  identical  to  the  four  models  executed  in  Janus. 

E.  EXPERIMENT 

1.  Conduct  of  Experiment 

Each  of  the  two  scenarios  (mechanized,  light)  was  executed  in  two  environments 
(Southwest  Asia,  Fort  Hunter-Liggett)  and  run  in  two  different  modes  (Janus,  HLA 
Warrior).  The  two  scenarios,  two  environments,  and  two  modes  resulted  in  a  total  of 
eight  combinations  for  the  experiment.  Each  of  the  combinations  was  executed  10  times. 
In  total,  80  runs  were  performed  to  provide  a  reasonable  data  sample  to  analyze  each 
MOP. 

The  method  of  analysis  was  to  compare  the  MOPs  resulting  from  a  specific 
number  of  Janus  runs  to  the  MOPs  resulting  from  the  same  number  of  HLA  Warrior  runs. 
Since  Janus  produces  exactly  the  same  results  when  the  same  random  number  seed  is 
used  (likewise  for  HLA  Warrior),  all  Janus  runs  were  executed  using  different  randomly 
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selected  seeds.  The  same  random  number  seeds  were  then  used  in  the  corresponding 
HLA  Warrior  runs.  While  the  random  number  generator  in  HLA  Warrior  is  intended  to 
be  identical  to  the  random  number  generator  in  Janus,  the  implementation  of  the 
algorithms  (discussed  in  chapter  4,  Statistical  Analysis),  caused  the  random  number 
draws  in  the  two  simulations  to  lose  synchronization  rapidly.  Despite  the  differences  in 
the  random  number  utilization,  the  design  of  the  experiment  supports  paired  output  from 
the  two  simulations. 

2.  Experimental  Design  Issues 

Determining  sample  size  is  fundamental  to  the  design  of  every  experiment.  An 
experiment  is  conducted  a  number  of  times  so  that  the  data  produces  good  estimators  of 
the  true  population  parameters.  Several  techniques  are  available  to  arrive  at  a  satisfactory 
sample  size. 

First,  experimenters  may  apply  a  practical  approach.  Based  on  experience  and 
recommendations  from  senior  Janus  analysts,  a  sample  size  of  n  =  10  generally  produces 
results  with  “acceptable  variance”  for  the  defined  MOPs.  That  is,  the  experiment  is  run 
until  an  estimate  of  the  variance  for  the  mean  is  reduced  to  a  pre-determined,  acceptable 
level.  The  acceptable  level  will  be  different  for  each  MOP. 

The  issue  of  normality  of  data  was  also  considered  when  determining  sample  size. 
Normality  of  the  data  set  is  one  assumption  required  prior  to  applying  the  paired  r-test 
(described  below).  Normal  probability  plots  were  used  to  test  for  approximate  normality. 
As  the  sample  size  increases,  one  would  expect  the  sample  averages  to  become  more 
normal,  based  on  the  Central  Limit  Theorem  [Ref.  4;p.  232].  Based  on  observations  of 
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normal  probability  plots  for  each  MOP,  a  sample  size  of  n  =  10  was  generally  considered 
large  enough  to  produce  normally  distributed  averages. 

F.  DATA  COLLECTION 

Data  was  collected  following  the  10  runs  of  each  of  the  eight  experimental 
designs  described  above.  Summary  statistics  for  each  MOP  were  computed  based  on  the 
raw  data  for  each  run  (see  Appendix  A).  Each  cell  entry  for  detection  and  kill  range  in 
Appendix  A  represent  an  average  of  all  detection  ranges  and  kill  ranges  for  the  specific 
scenario,  environment  and  mode  for  a  specific  scenario  run.  The  entries  for  Force 
Exchange  Ratio  and  Rounds  Fired  are  the  final  force  exchange  ratios  and  the  actual 
number  of  rounds  fired  for  each  specific  scenario  run  respectively.  Tables  1-3  below 
present  the  summary  statistics  used  in  the  analysis  of  each  MOP. 


Scenario 

Summary 

Statistic 

Detection  Range  Blue 

Detection  Range  Red 

Warrior 

Janus 

Warrior 

Janus 

HL  Light 

Average 

Standard  Deviation 

1304.230 

8.882 

1082.181 

28.961 

1359.080 

20.415 

1157.988 

9.096 

HL  Mech 

Average 

Standard  Deviation 

3527.878 

47.519 

4106.420 

221.948 

3071.678 

91.092 

3445.31 1 

200.412 

SWA  Light 

Average 

Standard  Deviation 

1480.730 

47.685 

1070.850 

45.207 

1796.850 

36.954 

1095.394 

26.454 

SWA  Mech 

Average 

Standard  Deviation 

3620.313 

314.325 

4469.942 

116.839 

2675.033 

201.431 

3721.506 

228.863 

Table  1.  Detection  Range  Summary  Statistics 
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Scenario 

Summary 

Statistic 

Warrior 

Janus 

Warrior 

Janus 

HL  Light 

Average 

Standard  Deviation 

1037.063 

43.028 

891 .740 
47.663 

1126.839 

36.116 

1044.613 

25.195 

HL  Mech 

Average 

Standard  Deviation 

2922.519 

126.787 

2920.349 

286.633 

2719.824 

109.508 

3153.435 

258.894 

SWA  Light 

Average 

Standard  Deviation 

979.520 

190.953 

658.122 

32.414 

987.069 

70.824 

806.023 

41.741 

SWA  Mech 

Average 

Standard  Deviation 

3448.491 

290.446 

3018.577 

194.556 

2258.870 

212.336 

2854.760 

286.861 

Table  2.  Kill  Range  Summary  Statistics 


Scenario 

Summary 

Statistic 

FER 

Rounc 

Is  Fired 

Warrior 

Janus 

Warrior 

Janus 

HL  Light 

Average 

Standard  Deviation 

0.187 

0.031 

0.171 

0.056 

6690.200 

425.022 

71883.100 

4316.057 

HL  Mech 

Average 

Standard  Deviation 

0.793 

0.143 

1.082 

0.480 

323.800 

33.963 

51 1 .500 

180.981 

SWA  Light 

Average 

Standard  Deviation 

0.326 

0.089 

18373.900 

1518.918 

27723.300 

4126.873 

SWA  Mech 

Average 

Standard  Deviation 

46.700 

11.615 

291.900 

49.983 

Table  3.  PER  and  Rounds  Fired  Summary  Statistics 
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IV.  STATISTICAL  ANALYSIS 


This  chapter  discusses  the  methodology  for  analyzing  the  results  and  the 
statistical  tests  used  in  the  analysis. 

A.  ANALYSIS  METHODOLOGY 

After  completing  all  runs  and  gathering  required  data,  rigorous  statistical  tests 
were  applied  to  determine  whether  HLA  Warrior  results  were  statistically  similar  to  Janus 
results.  For  each  simulation,  the  same  random  number  seed  was  used  in  both  Janus  and 
the  corresponding  HLA  Warrior  run.  The  only  difference  in  the  scenarios  was  the  mode 
(Janus  or  HLA  Warrior)  on  which  it  was  executed.  Since  identical  scenarios  were 
executed  in  identical  environments  on  opposing  simulations,  the  data  from  the  two 
simulations  is  paired.  Given  the  data  was  paired  between  Janus  and  HLA  Warrior,  two 
applicable  statistical  tests  were  used  to  determine  similarity:  the  Wilcoxon  Signed  Ranks 
Test  and  the  paired  r-test.  Initially,  all  MOPs  were  anal)^ed  using  the  Wilcoxon  Signed 
Ranks  test,  then  the  MOPs  that  satisfied  the  paired  t-test  normality  assumption  were 
analyzed  using  the  more  powerful  paired  t-test. 

B.  TOOLS  FOR  ANALYSIS 

1.  Wilcoxon  Signed  Ranks  Test 

As  the  actual  distributions  of  the  MOPs  are  not  known,  all  MOPs  were  first 
analyzed  using  a  pair-wise  comparison  by  means  of  the  non-parametric  Wilcoxon  Signed 
Ranks  Test.  The  Wilcoxon  Signed  Ranks  Test  begins  by  taking  the  data  consisting  of  n 
observed  pairs  {Xj.Yi),  (X2,Y2), ...,  {Xn,Yn),  and  computing  the  absolute  difference  (ID/I) 
between  each  of  the  n  pairs. 


15 


lAI  =  \Yi-Xi\ 


i  =  1,2,. ..,n 


Cases  where  the  differences  are  zero  (X,  equals  7,)  are  omitted.  The  number  of  remaining 
pairs  is  then  denoted  by  n\n ’<  n.  Ranks  from  1  to  n  ’  are  then  assigned  to  the  n ’  pairs 
according  to  the  relative  size  of  their  absolute  differences  as  follows.  Rank  1  is  given  to 
the  pair  (X/.T,)  with  the  smallest  absolute  difference  ID,I;  rank  2  is  given  to  the  pair  with 
the  second  smallest  difference,  and  so  on.  In  cases  of  a  tie,  the  average  of  the  ranks  that 
would  have  been  otherwise  assigned  is  assigned  to  each  of  the  pairs  in  the  tie. 

[Ref.  5:p.  206-207] 

First,  the  Wilcoxon  Signed  Ranks  Test  relies  on  several  assumptions  with  regard 
to  the  £),’s.  Di’s  are  assumed  to  be  continuous  and  independent.  These  assumptions  are 
justified  by  the  construction  of  the  A’s;  each  Z),  is  the  difference  of  two  real  numbers  and 
each  originates  as  a  result  of  independently  selected  random  number  seeds.  Another 
assumption  is  the  distribution  of  the  Z)/’s  is  assumed  to  be  symmetric.  If  the  two 
simulations  are  the  same  (as  they  are  under  the  null  hypothesis  that  the  HLA  Warrior  and 
Janus  populations  are  identical)  the  average  amount  by  which  Janus  MOPs  exceeds  HLA 
Warrior  MOPs  should  be  identical  to  the  average  amount  by  which  HLA  Warrior  MOPs 
exceeds  Janus  MOPs.  The  final  assumption  is  the  mean  of  the  distribution  of  the  D,’s  is 
some  hypothesized  value  (given  the  null  hypothesis,  the  value  of  the  mean  is  zero). 

[Ref.  5:p.  207] 

The  test  statistic,  T,  equals  the  sum  of  the  ranks  (R,)  assigned  to  those  pairs  where 
Yi  exceeds  X,-. 

Ri  =  0  if  Xi>Yi  {Di  negative). 

Ri  =  the  rank  assigned  to  (X„y,),  if  X,<y,  (Z),  positive). 
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Therefore,  the  test  statistic  is  given  by: 


T='ZRi. 

For  this  thesis,  the  null  hypothesis.  Ho,  is  that  the  mean  value  of  the  population  of 
X,’s  (HLA  Warrior  results  from  all  possible  random  number  seeds)  is  equal  to  the  value 
of  the  y,’s  (Janus  results).  The  alternate  hypothesis  is  that  the  HLA  Warrior  results  and 
the  Janus  results  are  not  equal.  The  null  hypothesis  is  rejected  in  favor  of  the  alternate 
hypothesis  at  the  level  of  significance  a  =  0.05  if: 

T  <  Wot/2  or  r  >  W\.aJ2- 

Critical  lower  and  upper  values  of  the  /?**’  quantile  [Ref.6:p.208],  with  a  =  0.05  and 
n=10,  are  9  and  46,  respectively  [Ref.  5:p.  383].  The  p-values  (the  smallest  level  of 
significance  at  which  the  null  h5q)othesis  would  be  rejected  when  a  specified  test 
procedure  is  used  [Ref.  4:p.  334])  for  the  test  statistic  for  each  MOP  was  then  determined 
using  the  S-Plus  statistical  software  package  [Ref.  6:p.  82]. 

2.  Paired  /-Test 

The  paired  r-test  relies  on  the  assumption  that  both  data  sets  to  be  compared  come 
from  normal  distributions.  Visual  observation  of  normal  probability  plots  of  each  of  the 
MOPs  was  used  to  determine  if  the  data  was  plausibly  normal.  For  those  cases 
determined  to  be  normal,  the  paired  /-test  was  applied.  As  discussed  previously  in 
chapter  three,  the  only  difference  in  each  run  was  the  mode  in  which  the  scenario  was 
run;  therefore  the  results  of  corresponding  HLA  Warrior  and  Janus  runs  are  paired  and 
utilization  of  the  paired  /-test  is  appropriate. 
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The  paired  Mest  takes  the  data  that  consist  of  n  observed  pairs  (Xj,Yj),  (X2,Y2), 
(X„,Y„),  and  computes  the  difference  (d/)  between  each  of  the  n  pairs.  For  this  thesis,  the 
X,’s  are  HLA  Warrior  MOP  results  and  the  F/’s  are  Janus  MOP  results.  The  mean  of  the 
n  differences  is  then  computed.  Under  the  null  hypothesis,  the  mean  of  the  differences 
for  each  of  the  MOPs  is  zero  in  the  population,  and  under  the  alternative  hypothesis,  the 
mean  of  the  differences  is  not  zero. 

Null  Hypothesis:  Ho'.  M-d  =  0 

Alternative  Hypothesis:  |1d  0 

The  test  statistic,  t,  for  the  analysis  is  defined  as: 

l4n 

where 

n 


and  the  sample  standard  deviation  is 

^  V  n-\ 

The  rejection  region  for  the  null  hypothesis  is: 

with  confidence  level  a  and  «  -  1  degrees  of  freedom.  If  t  falls  in  the  rejection  region, 
the  null  hypothesis  is  rejected  and  the  conclusion  is  that  HLA  Warrior  and  Janus  are  not 
statistically  similar.  [Ref.  4:p.  367-368]  For  this  thesis,  a  =  0.05. 
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C.  RESULTS 


1.  Wilcoxon  Signed  Ranks  Test 

The  Wilcoxon  Signed  Ranks  Test  indicates  that  HLA  Warrior  and  Janus  produce 
similar  results  in  only  a  few  instances.  Appendix  B  lists  the  results  of  the  Wilcoxon 
Signed  Ranks  Test.  Table  4  summarizes  the  results  for  each  MOP.  Only  three  of  the  24 
possible  outcomes  showed  that  HLA  Warrior  and  Janus  are  statistically  similar. 
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Not  Similar 

Not  Similar 
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Southwest  Asia 

Mech 

Not  Similar 

Not  Similar 

Not  Similar 

Not  Similar 

Not  Similar 

Not  Similar 

Table  4.  Wilcoxon  Signed  Ranks  Test  Results 


a.  Detection  Range 

In  all  environments  and  scenarios,  the  null  hypothesis  that  HLA  Warrior 
and  Janus  produced  similar  results  was  rejected  with  p-values  less  then  0.00195  in  every 
case.  Interestingly,  the  results  depended  heavily  on  the  scenario.  In  the  light  scenarios, 
HLA  Warrior  always  produced  detection  ranges  exceeding  Janus  detection  ranges. 
Conversely,  in  the  mechanized  scenarios,  Janus  always  produced  detection  ranges 
exceeding  HLA  Warrior  detection  ranges. 
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b.  Kill  Range 

In  one  of  the  eight  combinations  of  scenario  and  environment,  the  test 
leads  to  a  failure  to  reject  the  null  hypothesis  that  the  population  mean  kill  ranges 
produced  by  HLA  Warrior  and  Janus  are  identical.  The  blue  kill  ranges  in  the  Hunter- 
Liggett,  mechanized  scenario  were  found  to  be  statistically  indistinguishable  with  a  p- 
value  of  0.375.  In  all  other  combinations  the  null  hypothesis  was  rejected  with  p-values 
less  then  0.0058  in  each  case. 

c.  Force  Exchange  Ratio 

Only  the  Fort  Hunter-Liggett  scenarios  led  to  a  failure  to  reject  the  null 
hypothesis  that  the  force  exchange  ratios  produced  by  HLA  Warrior  and  Janus  are 
similar.  The  p-value  for  the  light  infantry  scenario  was  0.492  and  the  p-value  for  the 
mechanized  scenario  was  0.083.  The  null  hypothesis  was  rejected  for  both  scenarios, 
light  infantry  and  mechanized,  executed  in  Southwest  Asia  with  p-values  less  than  0.002. 

d.  Rounds  Fired 

In  all  cases,  the  null  hypothesis  that  HLA  Warrior  and  Janus  fire  the  same 
number  of  rounds  was  rejected  with  p-values  of  less  than  0.00391  in  each  instance  case. 
In  all  cases,  the  number  of  rounds  fired  by  Janus  was  much  greater  than  the  number  of 
rounds  fired  by  HLA  Warrior.  In  one  instance,  the  Hunter-Liggett  light  scenario,  Janus 
fired  an  order  of  magnitude  greater  than  HLA  Warrior. 

2.  Paired  r-Test 

After  the  analysis  using  the  Wilcoxon  Signed  Ranks  Test,  the  data  was  analyzed 
using  the  paired  r-test  for  those  data  sets  satisfying  conditions  of  normality.  The  first  step 
of  the  paired  r-test  was  to  determine  if  the  data  followed  a  normal  distribution  using 
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normal  probability  plots.  The  results  of  the  normality  evaluations  are  found  in  Table  5 
below.  The  normal  probability  plots  are  shown  in  Appendix  D. 
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Table  5.  Normal  Probability  Plot  Results 


Those  MOPs  found  to  be  normally  distributed  were  then  subjected  to  the  paired  t- 
test.  The  paired  r-test  confirms  the  Wilcoxon  Signed  Ranks  Test’s  findings  that  HLA 
Warrior  and  Janus  generally  do  not  produce  similar  output.  Appendix  C  contains  the 
paired  r-test  results.  Table  6  summarizes  the  results  of  the  paired  r-test. 
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NA 
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Table  6.  Paired  t-Test  Results 


a.  Detection  Range 

Similar  to  the  Wilcoxon  Signed  Ranks  Test,  in  all  combinations  of 
scenario  and  environment  the  null  hypothesis  that  HLA  Warrior  detection  range  is 
statistically  similar  to  Janus’  detection  range  was  rejected.  P-values  were  less  than 
0.000567  for  the  detection  range  in  every  case. 

b.  Kill  Range 

The  null  hypothesis  that  the  results  are  similar  was  rejected  in  all  cases  of 
kill  range  with  the  exception  of  the  Hunter-Liggett  mechanized  scenario,  which  resulted 
in  a  p-value  of  0.983.  P-values  for  the  cases  where  the  null  hypothesis  was  rejected 
ranged  from  1.84E-06  to  0.00219. 

c.  Force  Exchange  Ratio 

The  results  from  the  paired  r-test  for  Force  Exchange  Ratio  confirmed  the 
results  of  the  Wilcoxon  Signed  Ranks  Test.  In  both  scenarios  executed  in  the  Hunter- 
Liggett  environment,  the  null  hypothesis  was  not  rejected  with  p-values  of  0.529  and 
0.082  for  the  light  and  mechanized  scenarios,  respectively.  However,  the  Hunter-Liggett 
mechanized  scenario  failed  to  pass  the  normality  test  and  therefore  the  results  are  not 
reliable.  The  null  hypothesis  was  rejected  for  both  scenarios  executed  in  Southwest  Asia 
with  p-values  of  0.000153  and  3.88E-07  for  the  light  and  mechanized  scenarios, 
respectively. 

d.  Rounds  Fired 

The  paired  r-test  for  the  number  of  rounds  fired  also  confirmed  the 
Wilcoxon  Signed  Ranks  Test.  In  all  cases  the  null  hypothesis  that  the  number  of  rounds 
fired  in  both  simulations  was  similar  was  rejected.  The  p-values  ranged  from  0.01 18  for 
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the  Hunter-Liggett  mechanized  scenario  to  3.65E-12  for  the  Hunter-Liggett  light 
scenario. 

D.  ANALYSIS  OF  RESULTS 

The  results  of  the  Wilcoxon  Signed  Ranks  Test  and  the  paired  f-test  clearly 
indicate  that  HLA  Warrior  and  Janus  do  not  produce  statistically  similar  results  with 
respect  to  the  tested  MOPs.  With  few  exceptions  the  null  hypothesis  that  Janus  MOPs 
are  statistically  similar  to  HLA  Warrior  MOPs  was  rejected.  The  next  step  was  to 
analyze  why  differences  exist  and  identify  potential  causes.  Two  sources  of  the 
differences  investigated  were  HLA  Warrior  source  code  errors  and  differences  in  the 
implementation  of  the  Janus  algorithms  in  HLA  Warrior. 

1.  Source  Code  Errors 

Given  that  HLA  Warrior  is  a  new  simulation  still  undergoing  verification  and 
validation  testing,  a  starting  point  for  identifying  causes  to  the  disparate  results  was  the 
source  code  logic.  Analysis  of  the  code  began  with  the  search  and  detection  algorithms 
and  kill  algorithms.  An  error  in  the  detection  algorithm  would  likely  lead  to  errors  in  the 
other  MOPs.  For  instance,  changes  to  detection  range  would  likely  affect  the  kill  range 
since  these  MOPs  are  closely  linked.  Also,  the  number  of  rounds  fired  will  impact  the 
number  of  kills  and  potentially  kill  ranges.  Further  the  number  of  kills  will  ultimately 
affect  the  force  exchange  ratio  in  the  battle. 

Initial  investigation  into  the  search  and  detection  algorithm  and  the  kill  algorithm 
found  two  errors  in  the  HLA  Warrior  source  code.  The  first  error  was  in  the  inter-fire 
time  calculations.  In  Janus,  there  are  several  parameters  that  can  be  specified  for  each 
weapon  system.  Among  them  are  range,  rounds  per  trigger  pull,  trigger  pulls  per  reload. 
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and  reload  times.  Recall  all  scenarios  were  developed  in  Janus  then  converted  to  HLA 
Warrior.  As  HLA  Warrior  read  the  Janus  data,  a  conversion  error  occurred  when 
processing  the  number  of  trigger  pulls  per  reload.  Specifically,  HLA  Warrior 
exaggerated  the  inter-fire  times  for  systems  with  the  trigger  pulls  per  reload  parameter 
exceeding  one.  Therefore  the  overall  number  of  rounds  fired  by  HLA  Warrior  was  less 
than  expected. 

The  second  error  in  the  source  code  allowed  the  AH-64s  in  the  mechanized 
scenarios  to  detect  and  engage  targets  behind  them.  In  essence  the  AH-64s  fired  Hellfire 
missiles  backwards,  which  is  not  possible.  This  was  evident  while  watching  the 
mechanized  scenario  executed  in  Southwest  Asia.  In  each  run,  the  red  AH-64s  over  flew 
then  systematically  killed  the  blue  tanks.  As  a  result  the  blue  force  suffered  almost  total 
annihilation  while  inflicting  almost  no  casualties  on  the  red  force. 

HLA  Warrior  developers  corrected  both  errors  and  all  of  the  Hunter-Liggett  light 
infantry  scenarios  and  Southwest  Asia  mechanized  scenarios  were  re-executed.  Analysis 
of  the  results  showed  that  corrections  implemented  to  HLA  Warrior  source  code  lessened 
the  gap  between  Janus  and  HLA  Warrior  in  most  MOPs.  In  the  Southwest  Asia 
mechanized  scenario,  the  p-value  for  number  of  rounds  fired  changed  from  1 .02E-07 
prior  to  the  corrections  to  0.018  afterward.  Although  the  null  hypothesis  that  HLA 
Warrior  and  Janus  produce  similar  results  was  still  rejected,  the  gap  between  them  was 
significantly  reduced.  In  the  case  of  red  kill  ranges,  prior  to  implementing  the  changes, 
the  null  hypothesis  that  the  simulations  were  the  same  was  rejected  with  a  p-value  of 
0.0013.  After  the  changes,  the  null  hypothesis  is  no  longer  rejected  with  a  p-value  of 
0.108. 
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In  the  Hunter-Liggett  light  infantry  scenario,  the  disparity  between  Janus  and 
HLA  Warrior  was  reduced  in  four  of  the  six  MOPs,  blue  detection  range,  blue  kill  range, 
PER,  and  rounds  fired.  The  largest  improvement  was  in  the  number  of  rounds  fired. 
While  HLA  Warrior  still  fires  significantly  less  than  Janus,  the  magnitude  of  the 
difference  was  reduced  by  a  factor  of  10.  Tables  7  and  8  below  summarize  the  results  of 
the  Wilcoxon  Signed  Ranks  Test  and  the  paired  Mest  for  the  Hunter-Liggett  light  infantry 
and  the  Southwest  Asia  mechanized  scenarios  following  the  corrections  to  HLA  Warrior. 
Increased  p-values  indicate  greater  similarity  between  Janus  and  HLA  Warrior  results.  P- 
values  greater  than  0.05  fail  to  reject  the  null  hypothesis  that  the  results  are  statistically 
similar.  Appendix  E  contains  the  raw  data  and  analysis  output  for  the  comparison  of  and 
the  Hunter-Liggett  light  infantry  and  the  Southwest  Asia  mechanized  scenarios  utilizing 
the  corrected  version  of  HLA  Warrior. 
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Table  7.  Signed  Ranks  Test  Results  (p-Values)  Before  and  After  HLA  Warrior  Modifications 
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Table  8.  Paired  r-Test  Results  (p-Values)  Before  and  After  HLA  Warrior  Modifications 


While  the  modifications  reduced  the  disparity  between  Janus  and  HLA  Warrior, 
differences  still  exist  between  the  two  simulations.  In  all  of  the  30  light  infantry 
scenarios  (including  the  10  runs  following  the  corrections  to  the  programming),  HLA 
Wcirrior  entities  recorded  detection  events  at  a  greater  distance  than  Janus  entities. 
Conversely,  in  every  instance  in  the  mechanized  scenarios,  Janus  detection  ranges 
exceeded  HLA  Warrior  detection  ranges.  As  one  might  expect,  kill  ranges  tended  to 
follow  the  pattern  observed  in  the  detection  ranges.  In.  the  light  scenarios,  HLA  Warrior 
tended  to  have  longer  kill  ranges.  Conversely,  in  the  mechanized  scenarios,  Janus  tended 
to  have  longer  kill  ranges.  Therefore,  further  investigation  into  the  programming  may  be 
warranted. 

2.  Methodology  Differences 

Investigation  into  the  methodology  of  the  programming  of  HLA  Warrior  led  to 
several  possibilities  that  could  account  for  the  differences  between  the  two  simulations. 
Among  them  are  the  way  HLA  Warrior  reads  terrain  files  and  the  way  HLA  Warrior 
implements  some  Janus  algorithms. 
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a.  Terrain  Conversion 


As  mentioned  previously,  HLA  Warrior  possesses  the  capability  to  read 
scenario  files  constructed  in  Janus.  This  thesis  took  advantage  of  the  capability  to  ensure 
that  all  corresponding  scenarios  began  in  exactly  the  same  situation.  When  reading  entity 
locations  from  Janus,  HLA  Warrior  converts  all  Universal  Transverse  Macerator  (UTM) 
grid  locations  into  Latitude-Longitude  locations  [Ref.  7].  Due  to  the  inherent  errors  in 
the  conversion  process,  when  HLA  Warrior  positions  entities,  their  locations  may  be 
displaced  slightly  when  compared  to  their  original  locations  in  Janus.  This  displacement 
could  ultimately  effect  the  line  of  sight  for  those  entities.  In  hilly  or  rough  terrain  similar 
to  Fort  Hunter-Liggett,  entities  that  previously  had  clear  fields  of  view  in  Janus  may  be 
placed  behind  terrain  features  once  converted  to  HLA  Warrior,  inhibiting  line  of  sight. 
Reducing  the  line  of  sight  will  adversely  effect  detection  ranges. 

b.  Event  Scheduling 

While  HLA  Warrior  developers  took  great  efforts  to  ensure  that  the  Janus 
algorithms  remained  intact,  the  algorithms  were  often  implemented  differently  in  HLA 
Warrior.  The  primary  implementation  difference  is  in  the  way  events  are  scheduled  prior 
to  the  execution  of  the  individual  algorithms.  Janus  has  approximately  ten  major  events 
processing  at  regular  intervals.  Among  them  are  move  events,  search  events,  detection 
events,  and  impact  events.  As  the  simulation  clock  advances,  each  of  the  major  events  is 
processed  in  a  predetermined  priority.  The  search  event  is  executed  as  follows.  Prior  to 
starting  the  simulation  run,  the  user  can  specify  a  variable  called  dtsearch.  This  variable 
represents  the  time  required  for  an  entity  to  conduct  a  search.  For  instance,  let  dtsearch 
equal  6.  The  simulation  then  divides  dtsearch  by  the  number  of  sides  in  the  scenario  (say 
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2).  Each  side  in  the  scenario  is  then  assigned  an  equal  portion  of  dtsearch  to  conduct  its 
searches.  So,  for  example,  each  entity  on  side  one  conducts  its  first  search  at  time  zero. 
All  entities  on  side  one  then  conduct  the  next  search  dtsearch  time  units  later,  at  time  six 
in  this  example.  The  entities  on  side  two  conduct  their  first  search  at  time  three  in  this 
example.  Their  subsequent  search  will  commence  dtsearch  time  units  later,  at  time  nine. 
This  search  pattern  continues  throughout  the  duration  of  the  simulation.  Had  there  been 
three  sides,  the  first  side  would  have  conducted  searches  at  times  zero,  six,  twelve,  and  so 
on;  side  two  would  have  conducted  searches  at  time  two,  eight,  fourteen,  and  so  on;  and 
side  three  would  have  conduct  searches  at  times  four,  ten,  sixteen  and  so  on.  [Ref.  7] 

Each  search  event  then  stochastically  determines  if  a  future  detection 
event  will  be  scheduled  through  use  of  the  line-of-sight  and  detection  algorithms. 
Similarly,  detections  can  schedule  engagements,  and  subsequent  kill  events,  all  of  which 
are  executed  as  the  simulation  clock  advances.  [Ref.  7] 

HLA  Warrior  utilizes  the  same  major  events  as  Janus,  but  they  are 
implemented  differently.  In  the  case  of  the  search  event,  HLA  Warrior  also  utilizes  a 
variable  called  dtsearch,  defined  identically  to  the  Janus  variable.  However,  HLA 
Warrior  applies  a  different  method  to  determine  when  entities  conduct  searches.  Every 
entity  is  assigned  an  initial  search  time  according  to  a  uniform  random  variable  on  the 
interval  zero  to  dtsearch.  Recall  that  in  Janus,  all  entities  on  a  side  conduct  searches  at 
the  same  time.  The  second  search  for  each  HLA  Warrior  entity  takes  place  dtsearch  time 
units  after  the  first  search.  As  a  result  individual  entities  on  both  sides  are  conducting 
searches  throughout  the  dtsearch  time  period.  Consequently,  searches  by  both  sides  are 
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ongoing  continuously  for  the  duration  of  the  simulation.  Once  searches  commence,  the 
algorithm  to  determine  detections  is  identical  to  the  Janus  algorithm.  [Ref.  7] 

These  scheduling  difference  may  lead  to  differences  in  detections  and 
provide  a  partial  explanation  as  to  why  the  detection  ranges  in  the  HLA  Warrior  are 
different  from  the  detection  ranges  in  Janus.  Recall  that  in  Janus,  all  detections  for  a 
particular  side  occur  at  the  same  time,  then  are  repeated  dtsearch  time  units  later,  where 
as  in  HLA  Warrior,  detection  events  occur  continuously  throughout  the  simulation.  In 
the  time  between  search  events  in  Janus,  all  entities  will  have  the  opportunity  to  move 
greater  distances  than  HLA  Warrior  entities.  The  differences  in  entity  position  between 
each  detection  event  may  account  for  the  disparity  in  average  detection  range  between 
Janus  and  HLA  Warrior. 

While  the  event  scheduling  in  HLA  Warrior  is  different  from  the 
implementation  in  Janus,  HLA  Warrior  more  closely  models  reality.  In  an  actual  combat 
environment,  individual  soldiers  and  weapons  systems  are  not  limited  to  conducting 
searches  at  specific  times.  Rather,  they  conduct  searches  throughout  the  course  of  the 
battle.  Given  that,  HLA  Warrior’s  results  may  be  more  realistic  than  Janus’. 
c.  Line-of-Sight  Algorithm 

HLA  Warrior’s  line-of-sight  algorithm  has  also  been  modified  compared 
to  Janus’  line-of-sight  algorithm.  The  modifications  were  implemented  in  order  to  reduce 
the  overall  complexity,  and  thus  execution  time  of  the  algorithm.  In  basic  terms,  when 
determining  the  line  of  sight  between  two  entities,  Janus  first  determines  which  opposing 
entities  are  within  range  of  the  sensor  of  the  searching  entity.  For  those  entities  within 
range,  Janus  then  “walks  the  line”  between  the  entities  three  times.  The  first  time  it 
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walks  the  line,  Janus  collects  data  on  the  terrain  and  features  in  the  vicinity  of  the  line. 

On  the  second  walk  Janus  checks  to  determine  if  terrain  interferes  with  the  line  of  sight. 
The  third  walk  then  determines  if  any  features  in  the  vicinity  of  the  line  (trees,  buildings, 
etc)  interfere  with  the  line  of  sight.  These  calculations  are  then  computed  for  every  entity 
combination  on  the  battlefield.  [Ref.  7] 

HLA  Warrior,  on  the  other  hand,  only  walks  the  line  of  sight  twice.  On 
the  first  walk,  HLA  Warrior  determines  which  features  on  the  terrain  are  in  the  proximity 
of  the  line  of  sight  and  could  possibly  interfere  with  the  line.  On  the  second  walk,  both 
terrain  and  features  are  analyzed.  Additionally,  the  HLA  Warrior  designers  developed  a 
proximity  manager  that  reduces  the  number  of  entity  pairs  that  have  to  be  examined.  The 
proximity  manager  is  essentially  a  database  that  tracks  the  locations  of  all  entities  in  the 
simulation.  It  is  updated  after  the  execution  of  each  move  event.  When  determining  line 
of  sight,  the  searching  entity  queries  the  proximity  manager  as  to  which  entities  of  the 
opposing  force  are  within  its  sensor  search  pattern,  then  only  those  entity  pairs  within  the 
search  pattern  are  investigated  further  for  line  of  sight.  (The  proximity  manager  is  also 
utilized  in  artillery  casualty  assessment,  chemical  casualty  assessments,  and  movement 
delays  caused  by  obstacles  and  terrain  features).  Once  line  of  sight  is  determined,  the 
detection  and  kill  algorithms  in  HLA  Warrior  and  Janus  are  identical.  [Ref.  7] 

A  probable  difference  in  the  outcome  of  the  detection  algorithm  as  a  result 
of  implementation  differences  is  likely  in  the  proximity  manager.  When  determining  line 
of  sight,  Janus  looks  at  all  terrain  and  features  within  the  range  of  the  sensor,  while  HLA 
Warrior  looks  at  only  those  entities  passed  to  the  detection  algorithm  by  the  proximity 
manager.  Future  investigation  into  the  implementation  of  the  proximity  manager  is 
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required  to  determine  if  all  applicable  terrain  features  are  being  processed  for  line-of- 
sight  determination,  particularly  with  regard  to  mechanized  and  light  infantry  scenarios. 

The  net  result  of  these  different  implementations  is  that  HLA  Warrior 
tends  to  be  more  efficient  in  its  calculations  and  the  execution  of  the  algorithms. 
However,  the  implementation  of  the  algorithms  may  cause  the  disparate  results  in  the  two 
simulations. 
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V.  CONCLUSIONS 


The  initial  comparison  between  HLA  Warrior  and  Janus  indicated  that  the 
simulations  do  not  produce  statistically  similar  results  with  respect  to  most  of  the  selected 
MOPs.  In  almost  all  cases  the  null  hypothesis  that  the  simulations  were  the  same  was 
rejected.  Following  the  initial  analysis,  two  HLA  Warrior  programming  errors,  inter-fire 
times  between  reloads  and  helicopter  search  and  fire  sectors,  were  identified  and 
corrected.  The  subsequent  analysis  revealed  that,  while  the  simulations  were  still 
statistically  dissimilar,  some  differences  between  Janus  and  HLA  Warrior  were  reduced. 
Specifically,  differences  in  number  of  rounds  fired  and  kill  ranges  were  reduced  and  the 
FER  was  indistinguishable  following  the  corrections.  However,  consistent  differences 
remain  in  detection  ranges  between  Janus  and  HLA  Warrior. 

Certain  areas  in  HLA  Warrior  warrant  further  investigation.  First,  modification  of 
the  HLA  Warrior  search  sequence  to  match  Janus’  search  sequence  and  re-running  the 
scenarios  would  help  isolate  whether  HLA  Warrior’s  current  search  sequence  is  causing  a 
disparity  between  the  simulations.  Modifying  the  search  sequence  would  be  a  relatively 
straightforward  and  inexpensive  modification  to  HLA  Warrior.  Also,  follow-on  studies 
should  focus  on  examining  the  algorithms  in  both  simulations  to  determine  if  they  are  in 
fact  identical.  Finally,  additional  research  is  necessary  to  assess  HLA  Warrior’s  line-of- 
sight  algorithm.  Specifically  researchers  should  investigate  the  implementation  of  the 
proximity  manager  to  ensure  all  applicable  terrain  features  are  being  processed  for  line- 
of-sight  calculations. 
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Janus  is  considered  valid  primarily  through  user  confidence  and  face-validity. 
After  observing  40  scenarios  executed  in  Janus  and  the  same  scenarios  executed  60  times 
in  HLA  Warrior,  both  Janus  and  HLA  Warrior  scenarios  produce  generally  similar 
outcomes  and  the  HLA  Warrior  results  were  plausible.  Following  the  HLA  Warrior 
source  code  corrections,  the  entities  appeared  to  engage  and  kill  opposing  entities  at 
believable  distances  and  locations,  and  the  flow  of  the  battles  occurred  as  one  might 
expect.  Furthermore,  Janus  and  HLA  Warrior  did  produce  statistically  similar  force 
exchange  ratios,  the  quantitative  measure  of  the  scenario  outcome.  Consequently,  while 
HLA  Warrior  and  Janus  produce  statistically  different  detection  and  kill  ranges,  scenario 
outcomes  are  similar  and  HLA  Warrior  demonstrates  a  degree  of  face-validity. 
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APPENDIX  A  -  RAW  DATA 


SCENARIO  #1  HL  Light 


Detection  Range  Blue 

Detection  Range  Red 

wmsiissaism 

FER 

Rounds  Fired  I 

Run 

Warrior 

Janus 

Warrior 

Janus 

Warrior 

Janus 

Warrior 

Janus 

Warrior 

Janus 

Warrior 

Janus 

1 

1297.10 

1098.42 

1353.50 

1168.03 

1049.25 

1200.88 

1067.67 

0.217 

0.220 

6616 

72618 

2 

1311.90 

1113.54 

1388.80 

1146.07 

1060.41 

870.63 

1114.91 

1045.07 

0.154 

0.150 

66579 

3 

1307.00 

1080.47 

1359.20 

1155.76 

1107.67 

901.79 

1100.87 

1033.92 

0.178 

0.140 

77797 

4 

1296.00 

1042.76 

1338.90 

1174.89 

1037.43 

970.36 

1130.08 

1047.87 

0.238 

0.130 

68683 

5 

1309.30 

1063.36 

1375.90 

1156.48 

1032.46 

940.58 

1117.25 

1085.00 

0.167 

0.230 

69923 

6 

1301.40 

1106.17 

1326.80 

1152.94 

954.48 

904.74 

1061.21 

1062.74 

0.226 

0.090 

70233 

7 

1317.20 

1111.20 

1377.10 

1165.32 

1052.65 

853.86 

1157.30 

1045.92 

0.162 

0.270 

78171 

8 

1306.70 

1081.64 

1356.30 

1157.09 

992.80 

933.66 

1121.31 

1045.81 

0.152 

0.200 

76740 

9 

1308.70 

1029.20 

1376.70 

1146.74 

1072.54 

825.17 

1129.76 

1013.87 

0.178 

0.140 

67050 

10 

1287.00 

1095.05 

1337.60 

1156.57 

1010.95 

832.76 

1134.82 

998.27 

0.194 

0.140 

71037 

Average 

1304.23 

1082.18 

1359.08 

1157.99 

1037.06 

891.74 

1126.84 

1044.61 

0.187 

0.171 

6690.2 

71883.1 

Variance 

78.88 

838.73 

416.76 

82.73 

1851.44 

2271 .77 

1304.39 

634.77 

0.0009 

0.0031 

180643.3 

18628348.3 

Standard  Deviation 

8.88 

28.96 

20.41 

9.10 

43.03 

47.66 

36.12 

25.19 

0.0306 

0.0559 

425.0 

4316.1 

SCENARIO  #2  HLMech 


Detection  Range  Blue 

Detection  Range  Red 

■UULESaiSH 

FER 

Rounds  Fired  | 

Run 

Warrior 

Janus 

Warrior 

Janus 

Warrior 

Janus 

Warrior 

Janus 

Warrior 

Janus 

Warrior 

Janus 

1 

3546.19 

4077.79 

3092.52 

3243.25 

3008.90 

2603.06 

2756.89 

3099.05 

0.894 

0.740 

319 

657 

2 

3532.47 

3947.62 

3099.91 

3227.77 

2869.33 

2584.04 

2790.48 

3028.95 

0.628 

1.190 

275 

3 

3553.42 

4080.81 

3044.10 

3339.80 

2802.86 

2679.50 

3205.00 

0.821 

1.180 

315 

681 

4 

3482.95 

4512.96 

3062.95 

3686.48 

2830.76 

3526.67 

2607.39 

3475.63 

0.821 

0.410 

377 

441 

5 

3534.32 

4018.41 

3093.82 

3242.85 

2834.78 

2820.08 

2883.33 

0.773 

1.660 

337 

409 

6 

3612.19 

3896.11 

3109.91 

3387.61 

3096.67 

3001.56 

2808.21 

2857.86 

0.749 

1.390 

288 

296 

7 

3548.88 

4170.06 

3174.27 

3766.03 

3155.58 

3140.30 

2831.13 

3295.88 

0.556 

0.840 

287 

314 

8 

3444.83 

3900.53 

2949.61 

3402.75 

2832.18 

2753.80 

2497.97 

3076.25 

0.801 

1.360 

338 

749 

9 

3545.60 

3982.52 

3192.82 

3479.09 

2939.94 

2824.12 

2763.62 

2956.15 

1.087 

1.700 

368 

386 

10 

3477.94 

4477.38 

2896.87 

3677.49 

2799.30 

3132.31 

2642.98 

3656.25 

0.797 

0.350 

334 

419 

Average 

3527.88 

4106.42 

3071.68 

3445.31 

2922.52 

2920.35 

2719.82 

3153.43 

0.793 

1.082 

323.80 

511.50 

Variance 

2258.05 

49261.11 

8297.72 

40165.09 

16074.85 

82158.59 

11992.05 

67025.93 

0.0205 

0.2303 

1153.51 

32754.28 

Standard  Deviation 

47.52 

221.95 

91.09 

200.41 

126.79 

286.63 

109.51 

258.89 

0.1431 

0.4799 

33.96 

180.98 
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RAW  DATA 


SCENARIO  #3  SWA  Light 


Detection  Range  Blue 

Detection  Range  Red 

Kill  Range  Red 

FER  I 

Rounds  Fired 

Run 

Warrior 

Janus 

Warrior 

Janus 

Warrior 

Janus 

Warrior 

Janus 

Warrior 

Janus 

Warrior 

Janus 

1 

1481.00 

1059.99 

1822.90 

1081.07 

1200.82 

619.70 

1131.82 

852.95 

1.170 

0.380 

20243 

37716 

2 

1572.20 

1035.30 

1826.10 

1106.32 

931.31 

645.52 

956.81 

776.80 

0.577 

0.240 

?  17014 

25822 

3 

1512.00 

1135.79 

1803.40 

1138.70 

1281.35 

677.66 

923.04 

845.98 

0.887 

0.470 

16813 

31584 

4 

1458.70 

1089.99 

1757.00 

1088.31 

723.36 

704.71 

925.42 

796.24 

0.592 

0.370 

19167 

26326 

5 

1415.80 

1125.44 

1714.00 

1115.19 

812.65 

684.79 

1029.09 

824.82 

0.856 

0.210 

19668 

26138 

6 

1477.30 

1095.57 

1832.50 

1116.47 

985.64 

621.91 

1058.81 

855.80 

0.691 

0.250 

20857 

29079 

7 

•  1534.60 

995.47 

1803.10 

1089.60 

1062.25 

643.94 

939.87 

745.60 

0.821 

0-450 

16704 

26184 

8 

1460.40 

1051.76 

1796.30 

1043.45 

974.01 

671.62 

910.94 

748.53 

0.709 

0.280 

17365 

24654 

9 

1426.40 

1095.20 

1787.10 

1100.69 

1107.16 

693.43 

986.87 

782.86 

0.831 

0.300 

18376 

24379 

10 

1468.90 

1023.98 

1826.10 

1074.15 

716.65 

617.92 

1008.03 

830.67 

0.330 

0.310 

17532 

25351 

Average 

1480.73 

1070.85 

1796.85 

1095.39 

979.52 

658.12 

806.02 

0.746 

0.326 

18373.9 

27723.3 

Variance 

2273.85 

2043.70 

1365.57 

699.80 

36463.15 

1050.69 

1742.33 

0.0505 

0.0078 

2307111.7 

17031082.5 

Standard  Deviation 

47.68 

45.21 

36.95 

26.45 

190-95 

32.41 

41.74 

0.2247 

0.0886 

1518.9 

4126.9 

SCENARIO  #4  SWA  Mech 


Detection  Range  Blue 

Detection  Range  Red 

Kill  Range  Blue 

Kill  Range  Red 

FER 

Run 

Warrior 

Janus 

Warrior 

Janus 

Warrior 

Janus 

Warrior 

Janus 

Warrior 

Janus 

1 

3792,37 

4514.06 

2543.64 

3795.37 

3582.41 

3098.14 

2127.63 

2855.00 

0.121 

1.330 

41 

^  226 

2 

3709.24 

4524.34 

2603-43 

4135.74 

3124.60 

3218.29 

2052.46 

3293.75 

0.169 

2.220 

40 

293 

3 

3608.05 

4357.00 

2652.03 

3371.48 

3248.24 

2787.25 

2257.17 

2741.82 

0.145 

1.580 

51 

251 

4 

3924.61 

4337.02 

2858.22 

3486.64 

2900.43 

2842.25 

2521.61 

2248.13 

0.121 

1.080 

42 

327 

5 

3533.10 

4662.36 

2849.94 

3880.45 

3714.92 

3272.33 

2322.32 

3146.00 

0.121 

1.870 

74 

292 

6 

3424.19 

4509.96 

2801.35 

3768.72 

3731.13 

3145.95 

2149.47 

2908.75 

0.145 

2.280 

52 

346 

7 

3235.43 

4564.56 

2356.39 

3746.04 

3475.73 

3125.58 

2042.83 

2935.56 

0.121 

2.070 

40 

258 

8 

3534.35 

4270.55 

2566.15 

3540.80 

3618.89 

2708.78 

2291.12 

2889.17 

0.072 

1.480 

32 

362 

9 

3202.67 

4478.59 

2500.31 

3579.34 

3324.15 

2892.20 

2123.71 

2598.18 

0.121 

1.620 

42 

227 

10 

4239.12 

4480-99 

3018.88 

3910.49 

3764.42 

3095.00 

2700.39 

2931.25 

0.143 

2.170 

53 

337 

Average 

3620.31 

4469.94 

2675.03 

3721.51 

3448.49 

3018.58 

2258.87 

2854.76 

0.128 

1.770 

46.70 

291.90 

Variance 

98800.28 

13651-35 

40574.55 

52378.20 

84358-73 

37852.22 

45086.55 

82289.41 

0.0006 

0-1706 

134.90 

2498.32 

Standard  Deviation 

314.33 

116.84 

201.43 

228,86 

290.45 

194.56 

212.34 

286.86 

0-0255 

0-4130 

11.61 

49.98 
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APPENDIX  B  -  WILCOXON  SIGNED  RANKS  TEST  RESULTS 


Blue 

Blue 

Red 

Red 

Scenarrio 

Run 

Det  Range 
Warrior 

Det  Range 
Janus 

Difference 

Rank 

Det  Range 
Warrior 

Det  Range 
Janus 

Difference 

Rank 

HL  Light 

1 

1297.1 

1098.42 

198.7 

1353.5 

1168.028 

185.5 

HL  Light 

2 

1311.9 

1113.541 

198.4 

1388.8 

1146.065 

242.7 

HL  Light 

3 

1307.0 

1080.474 

226.5 

1359.2 

1155,756 

203.4 

HL  Light 

4 

1296.0 

253.2 

1338.9 

1174.892 

164.0 

HL  Light 

5 

1309.3 

245.9 

1375.9 

1156.483 

219.4 

HL  Light 

6 

1301 .4 

195.2 

1326.8 

1152.94 

173.9 

2 

HL  Light 

7 

1317.2 

206.0 

1377.1 

1165.323 

211.8 

7 

HL  Light 

8 

1306.7 

225.1 

1356.3 

1157.088 

199,2 

HL  Light 

9 

1308.7 

279.5 

10 

1376.7 

1146.741 

230.0 

HL  Light 

10 

1287.0 

192.0 

1 

1337.6 

1156.567 

181.0 

Average 

1304.23 

1082,18 

222.05 

55 

sum 

1359.08 

1157.9883 

201.0917 

55 

sum 

Variance 

Standard  Deviation 

78.885 

8.882 

838.727 

28.961 

0 

(9,46) 

p-Value 
Accept  region 

416.764 

20.415 

82.729 

9.096 

0 

(9.46) 

p-Value 
Accept  region 

Scenarrio 

Run 

Blue 

Det  Range 
Warrior 

Blue 

Det  Range 
Janus 

Difference 

Rank 

Red 

Det  Range 
Warrior 

Red 

Det  Range 
Janus 

Difference 

Rank 

HL  Mech 
HL  Mech 
HLMech 
HL  Mech 
HL  Mech 
HL  Mech 
HL  Mech 
HL  Mech 
HL  Mech 
HL  Mech 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

3546.2 

3532.5 
3553.4 

3482.9 

3534.3 
3612.2 

3548.9 

3444.8 

3545.6 

3477.9 

4077.791 

3947.62 

4080.812 

4512.963 

4018.409 

3896.107 

4170.064 

3900,532 

3982.519 

4477.379 

•531.6 

•415.2 

-527.4 

•1030.0 

-484,1 

-283.9 

-621.2 

-455.7 

-436.9 

-999.4 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

3092.5 

3099.9 
3044.1 

3062.9 

3093.8 

3109.9 
3174.3 

2949.6 

3192.8 

2896.9 

3243.252 

3227.77 

3339.795 

3686.477 

3242.853 

3387.609 

3766.03 

3402.746 

3479.088 

3677.49 

-150.7 

-127.9 

-295.7 

-623.5 

•149.0 

-277.7 

-591.8 

-453.1 

-286.3 

-780.6 

1 

Average 

Variance 

Standard  Deviation 

3527,878 

2258.053 

47.519 

4106.42 

49261.111 

221.948 

-578.54 

0 

0.001953125 

(9.46) 

sum 
p-Value 
Accept  region 

3071.6782 

8297.719 

91.092 

3445.311 

40165.087 

200.412 

•373.6328 

0 

0.001953125 

(9.46) 

•sum 
p-Value 
Accept  region 

Blue 

Blue 

Red 

Red 

Det  Range 

Det  Range 

Det  Range 

Det  Range 

Scenarrio 

Run 

Warrior 

Janus 

Rank 

Warrior 

Janus 

Difference 

SWA  Light 

1 

1481 .0 

421.0 

1822.9 

1081.068 

741.8 

SWA  Light 

2 

1572.2 

536.9 

1826.1 

1106.317 

719.8 

7 

SWA  Light 

3 

1512.0 

376.2 

1803.4 

1138.703 

664.7 

2 

SWA  Light 

4 

1458.7 

368.7 

1757.0 

1088.305 

668.7 

3 

SWA  Light 

5 

1415.8 

290.4 

1714.0 

1115.187 

598.8 

1 

SWA  Light 

6 

1477.3 

381.7 

1832.5 

1116.474 

716.0 

6 

SWA  Light 

7 

1534.6 

539.1 

10 

1803.1 

1089.595 

713.5 

5 

SWA  Light 

8 

1460.4 

408.6 

6 

1796,3 

1043.445 

752.9 

10 

SWA  Light 

9 

1426.4 

331.2 

2 

1787.1 

1100.693 

686.4 

4 

SWA  Light 

10 

1468.9 

444.9 

8 

1826.1 

1074.152 

751.9 

9 

Average 

1480.73 

1070.85 

409.88 

55 

sum 

1796,85 

1095.3939 

701.4561 

55 

sum 

Variance 

2273.847 

2043.700 

0 

p-Value 

1365.569 

699.803 

0 

p-Value 

Standard  Deviation 

47.685 

45.207 

(9.46) 

Accept  region 

36.954 

26.454 

(9.46) 

Accept  region 

Blue 

Blue 

Red 

Red 

Det  Range 

Det  Range 

Det  Range 

Det  Range 

Scenarrio 

Run 

Warrior 

Janus 

Difference 

Rank 

Warrior 

Janus 

Difference 

Rank 

SWA  Mech 

3792,4 

4514.058 

-721.7 

0 

2543.6 

3795.37 

-1251.7 

SWA  Mech 

3709,2 

4524.336 

-815.1 

0 

2603.4 

4135.74 

-1532.3 

SWA  Mech 

3608.0 

4356.996 

-748.9 

0 

'  2652.0 

3371.475 

•719.5 

SWA  Mech 

3924.6 

4337.016 

-412.4 

0 

2858.2 

3486.64 

•628.4 

SWA  Mech 

3533.1 

4662.361 

-1129.3 

0 

2849.9 

3880.451 

-1030.5 

SWA  Mech 

3424.2 

4509.962 

•1085.8 

0 

2801.3 

3768.718 

•967.4 

SWA  Mech 

3235.4 

4564.557 

-1329.1 

0 

2356.4 

3746.042 

-1389.7 

0 

SWA  Mech 

3534.3 

4270.554 

-736.2 

0 

2566.1 

3540.8 

•974.7 

SWA  Mech 

3202.7 

4478.59 

-1275.9 

0 

2500.3 

3579.338 

-1079.0 

SWA  Mech 

4239.1 

4480.992 

•241.9 

0 

3018.9 

3910.486 

•891 .6 

HoH 

Average 

3620.3128 

4469,94 

-849.63 

0 

sum 

2675.0325 

3721.506 

-1046.4735 

0 

sum 

Variance 

98800.277 

13651.354 

0.001953125 

p-Value 

40574.549 

52378.197 

0.001953125 

p-Value 

Standard  Deviation 

314.325 

116.839 

(9.46) 

Accept  region 

201.431 

228.863 

(9.46) 

Accept  region 

37 


WILCOXON  SIGNED  RANKS  TEST  RESULTS 


Scenario 

Run 

Blue 

Kill  Range 
Warrior 

Blue 

Kill  Range 
Janus 

Difference 

Rank 

Red 

Kill  Range 
Warrior 

Red 

Kill  Range 
Janus 

Difference 

Rank 

HL  Light 

1 

1049.25 

883.86 

165.39 

1200.88 

1067.67 

9 

HL  Ught 

2 

1060.41 

870.63 

189.79 

1114.91 

1045.07 

4 

HL  Ught 

3 

1107.67 

901.79 

205.88 

1100.87 

1033.92 

66.95 

3 

HL  Ught 

4 

1037.43 

970.36 

67.08 

1130.08 

1047.87 

82.22 

6 

HL  Light 

5 

1032.46 

940.58 

91.88 

1117.25 

1085.00 

32.25 

3 

HL  Ught 

6 

904.74 

49.74 

1061.21 

1062.74 

-1.53 

0 

HL  Ught 

1052.65 

853.86 

198.79 

1157.30 

1045.92 

111.38 

7 

HL  Ught 

992.80 

933.66 

59.14 

1121.31 

1045.81 

75.50 

5 

HL  Ught 

1072.54 

825.17 

247.37 

1129.76 

1013.87 

115.90 

8 

HL  Uqht 

1010.95 

832.76 

1  178.19 

HHHHI 

1134.82 

998.27 

136.55 

10 

Average 

1037.06 

891 .74 

145.32 

55 

sum 

1126.84 

1044,61 

82.23 

55 

sum 

Variance 

1851.44 

2271.77 

(9.46) 

accept  region 

1304.39 

634.77 

(9.46) 

accept  region 

Standard  Deviation 

43.03 

47.66 

0 

p- value 

36.12 

25.19 

0 

p-value 

Scenario 

Run 

Blue 

Kill  Range 
Warrior 

Blue 

Kill  Range 
Janus 

Difference 

Rank 

Red 

Kill  Range 
Warrior 

Red 

Kill  Range 
Janus 

Difference 

Rank 

HL  Mech 

1 

3008.90 

2603.06 

405.85 

9 

2756.89 

3099.05 

-342.16 

HL  Mech 

2 

2869.33 

2584.04 

285.30 

7 

2790.48 

3028.95 

-238.46 

HL  Mech 

3 

2791.31 

2802.86 

-11.54 

mm 

2679.50 

3205.00 

-525.50 

HL  Mech 

4 

2830.76 

3526.67 

-695.91 

2607.39 

3475.63 

-868.24 

HL  Mech 

5 

2901.20 

2834.78 

66.42 

2820.08 

2883.33 

-63.26 

HL  Mech 

6 

3096.67 

3001 .56 

95.11 

2808.21 

2857.86 

-49.65 

0 

HL  Mech 

7 

3155.58 

3140.30 

15.28 

2831.13 

3295.88 

-464.76 

HL  Mech 

8 

2832.18 

2753.80 

78.38 

2497.97 

3076.25 

•578.28 

HL  Mech 

9 

2939.94 

2824.12 

115.82 

2763.62 

2956.15 

-192.53 

HL  Mech 

10 

2799.30 

3132.31 

-333.00 

HeHI 

2642.98 

3656.25 

-1013.28 

Average 

2922.52 

2920.35 

2.17 

55 

sum 

2719.82 

3153.43 

-433.61 

55 

sum 

Variance 

16074.85 

82158.59 

(9.46) 

accept  region 

11992.05 

67025.93 

(9,46) 

accept  region 

Standard  Deviation 

126.79 

286.63 

0 

p-value 

109.51 

258.89 

0 

p-value 

Run 

Blue 

Kill  Range 
Warrior 

Blue 

Kill  Range 
Janus 

Difference 

Rank 

Red 

Kilt  Range 
Warrior 

Red 

Kill  Range 
Janus 

Difference 

Rank 

SWA  Light 

1 

1200.82 

619.70 

581.12 

9 

1131.82 

852.95 

278.87 

SWA  Ught 

2 

931.31 

645.52 

285.78 

4 

956.81 

776.80 

180.02 

SWA  Light 

3 

1281.35 

677.66 

603.69 

10 

923.04 

845.98 

77.06 

SWA  Ught 

4 

723.36 

704.71 

18.65 

1 

925.42 

796.24 

129.18 

SWA  Ught 

5 

812.65 

684.79 

127.86 

1029.09 

824.82 

204,27 

SWA  Ught 

6 

985.64 

621.91 

363.73 

1058.81 

855.80 

203.01 

SWA  Ught 

7 

1062.25 

643.94 

418.31 

939.87 

745.60 

194.27 

SWA  Ught 

8 

974.01 

671 .62 

302.39 

910.94 

748.53 

162.42 

SWA  Ught 

9 

1107.16 

693.43 

413.73 

986.87 

782.86 

204.01 

8 

SWA  Light 

10 

716.65 

617.92 

98.73 

1008.03 

830,67 

177.36 

mumi 

Average 

979.52 

658.12 

321.40 

55 

sum 

987.07 

806.02 

181.05 

55 

sum 

Variance 

36463.15 

1050.69 

(9.46) 

accept  region 

5016.07 

1742.33 

(9.46) 

accept  region 

Standard  Deviation 

190.95 

32.41 

0 

p-value 

.  70.82 

41.74 

0 

p-value 

Scenario 

Run 

Blue 

Kill  Range 
Warrior 

Blue 

Kill  Range 
Janus 

Difference 

Rank 

Red 

Kill  Range 
Warrior 

Red 

Kill  Range 
Janus 

Difference 

Rank 

SWA  Mech 

1 

3582.41 

3098.14 

484.27 

7 

2127.63 

2855.00 

-727.37 

0 

SWA  Mech 

2 

3124.60 

3218.29 

-93.70 

0 

2052.46 

3293.75 

-1241.29 

0 

SWA  Mech 

3 

3248.24 

2787.25 

460.99 

2257.17 

2741.82 

-484.65 

0 

SWA  Mech 

4 

2900.43 

2842.25 

58.18 

2521.61 

2248.13 

273.48 

2 

SWA  Mech 

5 

3714.92 

3272.33 

442.59 

2322.32 

3146.00 

-823.68 

0 

SWA  Mech 

6 

3731.13 

3145.95 

585.18 

2149.47 

2908.75 

-759.28 

SWA  Mech 

7 

3475.73 

3125.58 

350.15 

2042.83 

2935.56 

-892.73 

■■ 

SWA  Mech 

8 

3618.89 

2708.78 

910.11 

2291.12 

2889.17 

-598.04 

mm 

SWA  Mech 

9 

3324.15 

2892.20 

431.96 

2123.71 

2598.18 

-474.47 

0 

SWA  Mech 

10 

3764.42 

3095.00 

669.42 

2700.39 

2931.25 

-230.86 

0 

Average 

3448.49 

3018.58 

429.91 

55 

sum 

2258.87 

2854.76 

-595.89 

55 

sum 

Variance 

84358.73 

37852.22 

(9.46) 

accept  region 

45086.55 

82289.41 

(9.46) 

accept  region 

Standard  Deviation 

290.45 

194.56 

0 

p-value 

212.34 

286.86 

0 

p-value 
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Scenario 


HL  Light 
HL  Light 
HL  Light 
HL  Light 
HL  Light 
HL  Light 
HL  Light 
HL  Light 
HL  Light 
HL  Light 


Average 

Variance 

Standard  Deviation 


0.167 

0.226 

0.162 

0.152 

0.178 

0.194 


0.18662 

0.000936 

0.030601 


0.270 

0.200 

0.140 

0.140 


0.171 

0,003121 

0.055867 


Difference 


-0. 

0. 

0. 

0. 

-0. 

0. 

-0. 

-0. 

0. 

0. 


0.0156201 


sum 
P-Value 
accept  region 


Signed 

Rank 


0 

2 

3 

9 
0 

10 
0 
0 


6 


34 

0.4921875 

(9.46) 


Scenario 


SWA  Light 
SWA  Light 
SWA  Light 
SWA  Light 
SWA  Light 
SWA  Light 
SWA  Light 
SWA  Light 
SWA  Light 
SWA  Light 


PER  FER 

Warrior  Janus  Difference 


Signed 

Rank 


10 


Average 

Variance 

Standard  Deviation 


0.746192  0.326 

0.050468  0.007849 
0.224651  0.088594 


0.531 

0.020 


0.4201917 


sum 
P-Value 
accept  region 


Scenario 


HLMech 
HL  Mech 
HLMech 
HLMech 
HLMech 
HLMech 
HLMech 
HLMech 
HLMech 
HL  Mech 


Average 

Variance 

Standard  Deviation 


Scenario 

Run 

FER  FER 

Warrior  Janus  Difference 


0.821 

0.821 

0.773 

0.749 

0.556 

0.801 

1.087 

0.797 


0.792652 

0.020465 

0.143056 


Signed 

Rank 


1,082  -0.2893481 
0.230262 
0.479856 


1 

1 

7 

0 

3 

0 

4 

4 

10 

0 

9 

0 

2 

0 

6 

0 

8 

0 

5 

5 

sum 
P-Value 
accept  region 


10 

0.0839844 

(9.46) 


SWA  Mech 
SWA  Mech 
SWA  Mech 
SWA  Mech 
SWA  Mech 
SWA  Mech 
SWA  Mech 
SWA  Mech 
SWA  Mech 


Average 

Variance 

Standard  Deviation 


0.127833 

0.000648 

0.025458 


1.620 

2.170 


1.77 

0.170556 

0.412984 


-0.959 

-1.749 

-2.135 

-1.949 

-1.408 

-1.499 

-2.027 


1.6421671 


sum 
P-Value 
accept  region 


Signed 

Rank 


0 

0 

0 

0 

0 

0 

0 

0 

0 


0.0019531 

(9.46) 
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APPENDIX  C  -  PAIRED  /-TEST  RESULTS 


Detection  Range 


Hunter  Liggett  Light  Blue 
t-Test:  Paired  Two  Sample  for  Means 


Warrior 

Janus 

Mean 

1304.23 

1082.181 

Variance 

78.88456 

838.7271 

Observations 

10 

10 

Pearson  Correlation 

0.085404 

Hypothesized  Mean  Difference 

0 

df 

9 

tStat 

23.75604 

p(T<=t)  one-tail 

9.9E-10 

t  Critical  one-tail 

1.833114 

P(T<=:t)  two-tail 

1.98E-09 

t  Critical  two-tail 

2.262159 

Hunter  Liggett  Mech  Blue 

t*Test:  Paired  Two  Sample  for  Means 


Warrior 

Janus 

Mean 

3527.878 

4106.42 

Variance 

2258.053 

49261.11 

Observations 

10 

10 

Pearson  Correlation 

-0.429349 

Hypothesized  Mean  Difference 

0 

df 

9 

tStat 

-7.433366 

p(T<=t)  one-tail 

1.98E-05 

t  Critical  one-tail 

1.833114 

P(T<=:t)  two-tail 

3.96E-05 

t  Critical  two-tail 

2.262159 

Hunter  Liggett  Light  Red 

t-Test:  Paired  Two  Sample  for  Means 


Warrior 

Janus 

Mean 

1359.08 

1157.988 

Variance 

416.764 

82.72893 

Observations 

10 

10 

Pearson  Correlation 

-0.387218 

Hypothesized  Mean  Difference 

0 

df 

9 

t  Stat 

25.07204 

p(T<=t)  one-tail 

6.14E-10 

t  Critical  one-tail 

1.833114 

P(T<=t)  two-tail 

1.23E-09 

t  Critical  two-tail 

2.262159 

Hunter  Liggett  Mech  Red 

t-Test:  Paired  Two  Sample  for  Means 


Warrior 

Janus 

Mean 

3071.678 

3445.311 

Variance 

8297.719 

40165.09 

Observations 

10 

10 

Pearson  Correlation 

-0.089057 

Hypothesized  Mean  Difference 

0 

df 

9 

tStat 

-5.19564 

P(T<=t)  one-tail 

0.000284 

t  Critical  one-tail 

1.833114 

p(T<=t)  two-tail 

0.000567 

t  Critical  two-tail 

2.262159 
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Detection  Range 


SWA  Light  Blue 

t-Test:  Paired  Two  Sample  for  Means 


Warrior 

Janus 

Mean 

1480.73 

1070.85 

Variance 

2273.847 

2043.7 

Obsen/ations 

10 

10 

Pearson  Correlation 

-0.500798 

Hypothesized  Mean  Difference 

0 

df 

9 

tStat 

16.10573 

P(T<=t)  one-tail 

3.04E-08 

t  Critical  one-tall 

1.833114 

p(T<=t)  two-tail 

6.07E-08 

t  Critical  two-tail 

2.262150 

SWA  Mech  Blue 

t-Test:  Paired  Two  Sample  for  Means 


Warrior 

Janus 

Mean 

3620.313 

4469.942 

Variance 

98800.28 

13651.35 

Observations 

10 

10 

Pearson  Correlation 

-0.203658 

Hypothesized  Mean  Difference 

0 

df 

9 

tStat 

-7.527086 

P(T<=t)  one-tail 

1.79E-05 

t  Critical  one-tail 

1.833114 

P(T<=t)  two-tail 

3.59E-05 

t  Critical  two-tail 

2.262159 

SWA  Light  Red 

t-Test:  Paired  Two  Sample  for  Means 


Warrior 

Janus 

Mean 

1796.85 

1095.394 

Variance 

1365.569 

699.8031 

Observations 

10 

10 

Pearson  Correlation 

-0.120087 

Hypothesized  Mean  Difference 

0 

df 

9 

tStat 

46.25105 

P(T<=t)  one-tail 

2.58E-12 

t  Critical  one-tail 

1.833114 

p(T<=t)  two-tail 

5.17E-12 

t  Critical  two-tail 

2.262159 

SWA  Mech  Red 

t-Test:  Paired  Two  Sample  for  Means 


Warrior 

Janus 

Mean 

2675.033 

3721.506 

Variance 

40574.55 

52378.2 

Observations 

10 

10 

Pearson  Correlation 

0.147282 

Hypothesized  Mean  Difference 

0 

df 

9 

tStat 

-11.74603 

p(T<=t)  one-tall 

4.62E-07 

t  Critical  one-tall 

1.833114 

P(T<=t)  two-tail 

9.24E-07 

t  Critical  two-tail 

2.262159 
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Kill  Range 


Hunter  Liggett  Light  Blue 

t-Test:  Paired  Two  Sample  for  Means 


Warrior 

Janus 

Mean 

1037.063 

891.74 

Variance 

1851.439 

2271 .772 

Observations 

10 

10 

Pearson  Correlation 

-0.239252 

Hypothesized  Mean  Difference 

0 

df 

9 

t  Stat 

6.432146 

P(T<=t)  one-tail 

6.03E-05 

t  Critical  one-tail 

1.833114 

P(T<=t)  two-tail 

0.000121 

t  Critical  two-tail 

2.262159 

Hunter  Liggett  Mech  Blue 

t-Test:  Paired  Two  Sample  for  Means 


Warrior 

Janus 

Mean 

2922.519 

2920.349 

Variance 

16074.85 

82158.59 

Observations 

10 

10 

Pearson  Correlation 

0.0145 

Hypothesized  Mean  Difference 

0 

df 

9 

t  Stat 

0.022014 

P(T<=t)  one-tail 

0.491459 

t  Critical  one-tail 

1.833114 

p(T<=t)  two-tail 

0.982917 

t  Critical  two-tail 

2.262159 

Hunter  Liggett  Light  Red 

t-Test:  Paired  Two  Sample  for  Means 


Warrior 

Janus 

Mean 

1126.839 

1044.613 

Variance 

1304.389 

634.7711 

Observations 

10 

10 

Pearson  Correlation 

-0.001657 

Hypothesized  Mean  Difference 

0 

df 

9 

tStat 

5.90019 

P(T<=t)  one-tail 

0.000115 

t  Critical  one-tail 

1.833114 

P(T<=t)  two-tail 

0.000229 

t  Critical  two-tail 

2.262159 

Hunter  Liggett  Mech  Red 

t-Test:  Paired  Two  Sample  for  Means 


Warrior 

Janus 

Mean 

2719.824 

3153.435 

Variance 

11992.05 

67025.93 

Observations 

10 

10 

Pearson  Correlation 

-0.45706 

Hypothesized  Mean  Difference 

0 

df 

9 

tStat 

-4.232935 

p(T<=t)  one-tail 

0.001099 

t  Critical  one-tail 

1.833114 

P(T<=:t)  two-tail 

0.002197 

t  Critical  two-tail 

2.262159 
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Kill  Range 


SWA  Light  Blue 

t-Test:  Paired  Two  Sample  for  Means 


Warrior 

Janus 

Mean 

979.5198 

658.1215 

Variance 

36463.15 

1050.695 

Observations 

10 

10 

Pearson  Correlation 

-0.085774 

Hypothesized  Mean  Difference 

0 

df 

9 

t  Stat 

5.17472 

p(T<=:t)  one-tail 

0.000292 

t  Critical  one-tail 

1.833114 

P(T<=t)  two-tail 

0.000583 

t  Critical  two-tail 

2.262159 

SWA  Mech  Blue 

t"Test:  Paired  Two  Sample  for  Means 


Warrior 

Janus 

Mean 

3448.491 

3018.577 

Variance 

84358.73 

37852.22 

Observations 

10 

10 

Pearson  Correlation 

0.358888 

Hypothesized  Mean  Difference 

0 

df 

9 

tStat 

4.757753 

P(T<=t)  one-tail 

0.000516 

t  Critical  one-tail 

1.833114 

P(T<=:t)  two-tail 

0.001033 

t  Critical  two-tail 

2.262159 

SWA  Light  Red 

t-Test:  Paired  Two  Sample  for  Means 


Warrior 

Janus 

Mean 

987.0693 

806.0225 

Variance 

5016.073 

1742.328 

Observations 

10 

10 

Pearson  Correlation 

0.670276 

Hypothesized  Mean  Difference 

0 

df 

9 

tStat 

10.82863 

P(T<=t)  one-tail 

9.19E-07 

t  Critical  one-tail 

1.833114 

P(T<=t)  two-tail 

1.84E-06 

t  Critical  two-tail 

2.262159 

SWA  Mech  Red 

t-Test:  Paired  Two  Sample  for  Means 


Warrior 

Janus 

Mean 

2258.87 

2854.76 

Variance 

45086.55 

82289.41 

Observations 

10 

10 

Pearson  Correlation 

-0.337755 

Hypothesized  Mean  Difference 

0 

df 

9 

tStat 

-4.590263 

P(T<=t)  one-tail 

0.000654 

t  Critical  one-tail 

1.833114 

p(T<=t)' two-tail 

0.001309 

t  Critical  two-tail 

2.262159- 
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Force  Exchange  Ratio 


Hunter  Liggett  Light 

t-Test:  Paired  Two  Sample  for  Means 


Warrior 

Janus 

Mean 

0.18662 

0.171 

Variance 

0.000936 

0.003121 

Observations 

10 

10 

Pearson  Correlation 

-0.478422 

Hypothesized  Mean  Difference 

0 

df 

9 

tStat 

0.654641 

P(T<=t)  one-tail 

0.264536 

t  Critical  one-tail 

1.833114 

P(T<=t)  two-tail 

0.529072 

t  Critical  two-tail 

2.262159 

SWA  Light 

t-Test:  Paired  Two  Sample  for  Means 


Warrior 

Janus 

Mean 

0.746192 

0.326 

Variance 

0.050468 

0.007849 

Observations 

10 

10 

Pearson  Correlation 

0.32288 

Hypothesized  Mean  Difference 

0 

df 

9 

t  Stat 

6.23177 

P(T<=t)  one-tail 

7.65E-05 

t  Critical  one-tail 

1.833114 

P(T<=t)  two-tail 

0.000153 

t  Critical  two-tall 

2.262159 

Hunter  Liggett  Mech 

t-Test:  Paired  Two  Sample  for  Means 


Warrior 

Janus 

Mean 

0.792652 

1.082 

Variance 

0.020465 

0.230262 

Observations 

10 

10 

Pearson  Correlation 

0.234107 

Hypothesized  Mean  Difference 

0 

df 

9 

tStat 

-1.957084 

P(T<=t)  one-tail 

0.041014 

t  Critical  one-tail 

1.833114 

p(T<=t)  two-tail 

0.082028 

t  Critical  two-tail 

2.262159 

SWA  Mech 


t-Test:  Paired  Two  Sample  for  Means 

Warrior 

Janus 

Mean 

0.127833 

1.77 

Variance 

0.000648 

0.170556 

Obsen/ations 

10 

10 

Pearson  Correlation 

0.553716 

Hypothesized  Mean  Difference 

0 

df 

9 

tStat 

-13.00034 

P(T<=t)  one-tail 

1.94E-07 

t  Critical  one-tail 

1.833114 

P(T<=t)  two-tail 

3.88E-07 

t  Critical  two-tail 

2.262159 
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Rounds  Fired 


Hunter  Liggett  Light 

t-Test:  Paired  Two  Sample  for  Means 


Warrior 

Janus 

Mean 

6690.2 

71883.1 

Variance 

180643.3 

18628348 

Observations 

10 

10 

Pearson  Correlation 

0.11606 

Hypothesized  Mean  Difference 

0 

df 

9 

tStat 

-48.08282 

P(T<=t)  one-tail 

1.82E-12 

t  Critical  one-tail 

1.833114 

P(T<=t)  two-tail 

3.65E-12 

t  Critical  two-tail 

2.262159 

SWA  Light 

t-Test:  Paired  Two  Sample  for  Means 


Warrior 

Janus 

Mean 

18373.9 

27723.3 

Variance 

2307112 

17031082 

Observations 

10 

10 

Pearson  Correlation 

0.409474 

Hypothesized  Mean  Difference 

0 

df 

9 

tStat 

-7.844541 

P(T<=t)  one-tail 

1.29E-05 

t  Critical  one-tail 

1.833114 

P(T<=t)  two-tail 

2.59E-05 

t  Critical  two-tail 

2.262159 

Hunter  Liggett  Mech 

t-Test:  Paired  Two  Sample  for  Means 


Warrior 

Janus 

Mean 

323.8 

511.5 

Variance 

1153.511 

32754.28 

Observations 

10 

10 

Pearson  Correlation 

-0.139225 

Hypothesized  Mean  Difference 

0 

df 

9 

tStat 

-3.145005 

P(T<=t)  one-tail 

0.005916 

t  Critical  one-tail 

1.833114 

p(T<=t)  two-tail 

0.011832 

t  Critical  two-tail 

2.262159 

SWA  Mech 

t-Test:  Paired  Two  Sample  for  Means 

Warrior 

Janus 

Mean 

46.7 

291.9 

Variance 

134.9 

2498.322 

Observations 

10 

10 

Pearson  Correlation 

0.019656 

Hypothesized  Mean  Difference 

0 

df 

9 

tStat 

-15.17634 

p(T<=t)  one-tail 

5.1E-08 

t  Critical  one-tail 

1.833114 

P(T<=:t)  two-tail 

1.02E-07 

t  Critical  two-tail 

2.262159 
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NORMAL  PROBABILITY  PLOTS 


SWA  Light  Red  Detection  Ranges 


SWA  Light  Red  Oetecth 


APPENDIX  E  -  RAW  DATA  AND  ANALYSIS  RESULTS 
FOLLOWING  HLA  WARRIOR  SOURCE  CODE  CORRECTIONS 

Southwest  Asia  Mechanized 
(Wilcoxon  Signed  Ranks  Test  Results) 


Blue  Detection  Range  _  _ Red  Detection  Range 


run 

HLA  Warrior 

Janus 

difference 

rank 

HLA  Warrior 

Janus 

difference 

rank 

1 

3668.40 

4514.06 

-845.66 

0 

3174.60 

3795.37 

0 

2 

3983.60 

4524.34 

-540.74 

0 

3352.60 

4135.74 

-783.14 

0 

3 

4006.90 

4357.00 

-350.10 

0 

3213.90 

3371.48 

-157.58 

0 

4 

4000.30 

4337.02 

-336.72 

0 

3370.70 

3486.64 

-115.94 

0 

5 

3904.80 

4662.36 

-757.56 

0 

3297.70 

3880.45 

-582.75 

0 

6 

3917.70 

4509.96 

-592.26 

0 

3070.10 

3768.72 

-698.62 

0 

7 

3799.70 

4564.56 

-764.86 

3164.60 

3746.04 

-581.44 

0 

8 

3778.10 

4270.55 

-492.45 

3133.50 

3540.80 

-407.30 

0 

9 

3930.40 

4478.59 

-548.19 

3217.70 

3579.34 

-361.64 

0 

10 

3943.50 

4480.99 

-537.49 

331 1 .60 

3910.49 

-598.89 

0 

Average 

3893.34 

4469.94 

-576.60 

0 

Sum 

3230.70 

3721 .51 

-490.81 

0 

Sum 

Variance 

Standard  Deviation 

12193.79 

110.43 

13651.35 

116.84 

(9.46) 

0.001953 

Accept  Region 
P-Va!ue 

9855.70 

99.28 

52378.20 

228.86 

(9.46) 

0.001953 

Accept  Region 
P-Value 

Blue  Kill  Ranges 

run 

HLA  Warrior 

Janus 

difference 

IBSSSH 

Janus 

difference 

rank 

1 

3205.10 

3098.14 

106.96 

2 

3274.30 

2855.00 

419.30 

5 

2 

3264.00 

3218.29 

45.71 

1 

2850.80 

3293.75 

-442.95 

0 

3 

3263.00 

2787.25 

475.75 

7 

2653.30 

2741.82 

-88.52 

0 

4 

3111.90 

2842.25 

269.65 

5 

2261 .40 

2248.13 

13.28 

1 

5 

3431 .40 

3272.33 

3 

2539,40 

3146.00 

-606.60 

0 

6 

2890.60 

3145.95 

-255.35 

0 

2751.90 

2908.75 

-156.85 

0 

7 

3450.70 

3125.58 

325.12 

2476.90 

2935.56 

-458.66 

0 

8 

3550.70 

2708.78 

841.92 

3001.00 

2889.1 7 

111.83 

4 

9 

3372.80 

2892.20 

480.61 

2488.40 

2598.18 

-109,78 

0 

10 

3654.00 

3095.00 

559.00 

2456.10 

2931 .25 

-475.15 

0 

Average 

3319.42 

3018.58 

300.84 

51 

Sum 

2675.35 

2854.76 

-179.41 

10 

Sum 

Variance 

49370.27 

37852.22 

(9,46) 

Accept  Region 

90462.01 

82289.41 

(9,46) 

Accept  Region 

Standard  Deviation 

222.19 

194.56 

0.009766 

P-Value 

300.77 

286.86 

0.083984 

P-Value 

PER 

Rounds  Fired 

run 

HLA  Warrior 

Janus 

difference 

HE!i!Bi 

HLA  Warrior 

rank 

1 

0.181 

-1.149 

0 

367 

226 

141 

3 

2 

0.334 

-1.886 

0 

466 

293 

173 

6 

3 

0.339 

-1.241 

0 

694 

251 

443 

9 

4 

0.286 

-0.794 

0 

424 

327 

97 

1 

5 

0.453 

1,870 

-1.417 

0 

572 

292 

280 

8 

6 

0.381 

2.280 

-1.899 

0 

487 

346 

141 

4 

7 

0.238 

-1.832 

0 

529 

258 

271 

7 

8 

0.215 

1.480 

-1 .265 

0 

1545 

362 

1183 

10 

9 

0.191 

1.620 

-1.429 

0 

381 

227 

154 

5 

10 

0.215 

2.170 

-1 .955 

0 

436 

337 

99 

2 

Average 

0.283 

-1.487 

0 

Sum 

590 

292 

298 

55 

Sum 

Variance 

0.008 

0.171 

(9,46) 

Accept  Region 

121950 

2498 

(9.46) 

Accept  Region 

Standard  Deviation 

0.091 

0.413 

0.001953 

P-Value 

349 

50 

0 

P-Value 

RAW  DATA  AND  ANALYSIS  RESULTS  FOLLOWING  HLA 
WARRIOR  SOURCE  CODE  CORRECTIONS 

Southwest  Asia  Mechanized 
(Paired  t-Test  Results) 


t-Test:  Paired  Two  Sample  for  Means 

Blue  Detection  Range  After  Warrior  Modifications 


HLA  Warrio 

Janus 

Mean 

3893.34 

4469.942 

Variance 

12193.79 

13651.35 

Observations 

10 

10 

Pearson  Correlation 

-0.119579 

Hypothesized  Mean  Difference 

0 

df 

9 

tStat 

-10.72004 

P(T<=t)  one-tail 

IE-06 

t  Critical  one-tail 

1.833114 

P(T<=t)  two-tail 

2E-06 

t  Critical  two-tail 

2.262159 

t-Test:  Paired  Two  Sample  for  Means 

Blue  Kill  Range  After  Warrior  Modifications 

HLA  Warrio 

Janus 

Mean 

3319.42 

3018.577 

Variance 

49370.27 

37852.22 

Observations 

10 

10 

Pearson  Correlation 

-0.091261 

Hypothesized  Mean  Difference 

0 

df 

9 

tStat 

3.084757 

P(T<=t)  one-tail 

0.00652 

t  Critical  one-tail 

1.833114 

P(T<=t)  two-tail 

0.013039 

t  Critical  two-tail 

2.262159 

t-Test:  Paired  Two  Sample  for  Means 

PER  After  Warrior  Modifications 

HLA  Warrio 

Janus 

Mean 

0.2833 

1.77 

Variance 

0.008266 

0.170556 

Observations 

10 

10 

Pearson  Correlation 

0.340082 

Hypothesized  Mean  Difference 

0 

df 

9 

tStat 

-12.00814 

p(T<=t)  one-tail 

3.83E-07 

t  Critical  one-tail 

1.833114 

P(T<=t)  two-tail 

7.66E-07 

t  Critical  two-tail 

2.262159 

t-Test:  Paired  Two  Sample  for  Means 

Red  Detection  Range  After  Warrior  Modifications 


HLA  Warrio 

Janus 

Mean 

3230.7 

3721.506 

Variance 

9855.698 

52378.2 

Observations 

10 

10 

Pearson  Correlation 

0.2714 

Hypothesized  Mean  Difference 

0 

df 

9 

tStat 

-6.947913 

P(T<=t)  one-tail 

3.35E-05 

t  Critical  one-tall 

1.833114 

P(T<=t)  two-tail 

6.7E-05 

t  Critical  two-tail 

2.262159 

t-Test;  Paired  Two  Sample  for  Means 

Red  Kill  Range  After  Warrior  Modifications 

HLA  Warrio 

Janus 

Mean 

2675.35 

2854.76 

Variance 

90462.01 

82289.41 

Observations 

10 

10 

Pearson  Correlation 

0.414261 

Hypothesized  Mean  Difference 

0 

df 

9 

tStat 

-1.782835 

P(T<=t)  one-tail 

0.054145 

t  Critical  one-tail 

1.833114 

P(T<=t)  two-tail 

0.10829 

t  Critical  two-tail 

2.262159 

t-Test:  Paired  Two  Sample  for  Means 

Rounds  Fired  After  Warrior  Modifications 

HLA  Warrio 

Janus 

Mean 

590.1 

291.9 

Variance 

121950.3 

2498.322 

Observations 

10 

10 

Pearson  Correlation 

0.474476 

Hypothesized  Mean  Difference 

0 

df 

9 

tStat 

2.870962 

p(T<=t)  one-tail 

0.009224 

t  Critical  one-tail 

1.833114 

P(T<=t)  two-tail 

0.018448 

t  Critical  two-tail 

2.262159 
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RAW  DATA  AND  ANALYSIS  RESULTS  FOLLOWING  HLA 
WARRIOR  SOURCE  CODE  CORRECTIONS 

Hunter-Liggett  Light  Infantry 
(Wilcoxon  Signed  Ranks  Test  Results) 


Blue  Detection  Range 

Red  Detection  Range 

.... 

run 

HLA  Warrior 

Janus 

HLA  Warrior 

Janus 

difference 

rank 

1 

HSU 

1098.42 

228.45 

227.69 

6 

2 

1113.54 

198.01 

BH 

HB 

221.43 

3 

3 

Bm 

1080.47 

229.74 

BB 

HH 

223.66 

4 

BB 

1042.76 

254.79 

BH 

171.11 

^b^h 

5 

BB 

1063.36 

271.64 

BB 

HH 

249.75 

6 

BH^ 

1106.17 

215.04 

BOB 

249.40 

7 

mmm 

1111.20 

196.08 

2 

BB 

BH 

225.92 

8 

1081.64 

246.81 

BH 

BH 

251.09 

10 

9 

1029.20 

282.24 

HH 

BB 

HH 

246.28 

7 

10  • 

1095.05 

188.61 

204.56 

2 

Average 

1082.18 

55 

Sum 

1385.08 

1157.99 

55 

Sum 

Variance 

838.73 

(9,46) 

Accept  Region 

437.26 

82.73 

(9.46) 

Accept  Region 

Standard  Deviation 

28.96 

0 

P-Value 

20.91 

9.10 

0 

P-Value 

Blue  Kill  Ranges 

Red  Kill  Ranges 

run 

HLA  Warrior 

Janus 

BES2H 

HLA  Warrior 

Janus 

difference 

I^^BI 

1 

■HM 

883.86 

177.74 

5 

1133.75 

1067.67 

66.08 

2 

2 

BH 

870.63 

197.51 

8 

1139.72 

1045.07 

94.65 

6 

3 

BH 

901.79 

160.78 

4 

1147.54 

1033.92 

113.62 

8 

4 

BH 

970.36 

‘81.66 

0 

1101.10 

1047.87 

53.23 

1 

5 

BH 

940.58 

180.52 

6 

1169.61 

1085.00 

84.61 

6 

BH 

904.74 

158.23 

3 

1158.35 

1062.74 

95.61 

7 

BH 

853.86 

188.98 

7 

1130.61 

1045.92 

84.69 

8 

BH 

933.66 

118.91 

2 

1125.77 

1045.81 

79.96 

9 

BH 

825.17 

269.21 

9 

1155.77 

1013.87 

141.90 

10 

832.76 

00 

10 

1161.56 

998.27 

10 

Average 

1063.61 

891.74 

54 

Sum 

1142.38 

1044.61 

Sum 

Variance 

Standard  Deviation 

5472.45 

73.98 

2271.77 

47.66 

(9,46) 

0.00195 

Accept  Region 
P'Value 

418.92 

20.47 

634.77 

25.19 

Hi 

Accept  Region 
P-Vatue 

PER 

Rounds  Fired 

run 

HLA  Warrior 

Janus 

HSil^l 

HLA  Warrior 

Janus 

difference 

rank 

1 

0.14 

0.22 

15686.00 

72618.00 

-56932.00 

0 

2 

0.20 

0.15 

16784.00 

66579.00 

-49795.00 

0 

3 

0.12 

0.14 

14669.00 

-63128.00 

0 

4 

0.42 

0.13 

22918.00 

68683.00 

-45765.00 

0 

5 

0.16 

0,23 

12718.00 

-57205.00 

0 

6 

0.20 

0.09 

0.11 

9 

15238.00 

70233.00 

-54995.00 

0 

7 

0.19 

0.27 

-0.08 

0 

17177.00 

78171.00 

-60994.00 

0 

8 

0.21 

0.20 

0.01 

1 

14598.00 

76740.00 

-62142.00 

0 

9 

0.16 

0-14 

0.02 

2 

13613.00 

67050.00 

-53437.00 

0 

10 

0.11 

0.14 

-0.03 

0 

15001.00 

71037.00 

-56036.00 

0 

Average 

0.19 

0.17 

27 

Sum 

15840.20 

71883.10 

BHH| 

0 

Sum 

Variance 

0.01 

0.00 

(9.46) 

Accept  Region 

7943005.29 

18628348.32 

(9.46) 

Accept  Region 

Standard  Deviation 

0.09 

0.06 

0.5 

P-Value 

2818.33 

4316.06 

HHh 

0.00195 

P-Value 
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RAW  DATA  AND  ANALYSIS  RESULTS  FOLLOWING  HLA 
WARRIOR  SOURCE  CODE  CORRECTIONS 

Hunter-Liggett  Light  Infantry 
(Paired  /-Test  Results) 


Hunter-Liggett  Light  Blue  Detection  Range 

t-Test;  Paired  Two  Sample  for  Means 


HLA  Warrior 

Janus 

Mean 

1313.321 

1082.181 

Variance 

236.2907433 

838.7271 

Observations 

10 

10 

Pearson  Correlation 

0.026785455 

Hypothesized  Mean  Difference 

0 

df 

9 

tStat 

22.54443499 

P(T<=:t)  one-tail 

1.57506E-09 

t  Critical  one-tail 

1.833113856 

P(T<=t)  two-tail 

3.15012E-09 

t  Critical  two-tail 

2.262158887 

Hunter-Liggett  Light  Blue  Kill  Range 

t-Test:  Paired  Two  Sample  for  Means 


HLA  Warrior 

Janus 

Mean 

1063.606 

891.74 

Variance 

5472.449471 

221^112 

Observations 

10 

10 

Pearson  Correlation 

-0.62083907 

Hypothesized  Mean  Difference 

0 

df 

9 

tStat 

4.936256062 

p(T<=t)  one-tail 

0.000403188 

t  Critical  one-tail 

1.833113856 

p(T<=:t)  two-tail 

0.000806376 

t  Critical  two-tail 

2.262158887 

Hunter-Liggett  Light  PER 

t-Test:  Paired  Two  Sample  for  Means 

HLA  Warrior 

Janus 

Mean 

0.1908 

0.171 

Variance 

0.007399956 

0.003121 

Observations 

10 

10 

Pearson  Correlation 

-0.17982756 

Hypothesized  Mean  Difference 

0 

df 

9 

tStat 

0.565725575 

P{T<=t)  one-tail 

0.292703215 

t  Critical  one-tail 

1.833113856 

p(T<=t)  two-tail 

0.58540643 

t  Critical  two-tail 

2.262158887 

Hunter-Liggett  Light  Red  Detection  Range 

t“Test:  Paired  Two  Sample  for  Means 


HLA  Warrior 

Janus 

Mean 

1385.077 

1157.988 

Variance 

437.2590456 

82.72893 

Observations 

10 

10 

Pearson  Correlation 

-0.28114945 

Hypothesized  Mean  Difference 

0 

df 

9 

tStat 

28.68035701 

p(T<=t)  one-tail 

1.85458E-10 

t  Critical  one-tail 

1.833113856 

P(T<=t)  two-tail 

3.7091 5E-10 

t  Critical  two-tai! 

2.262158887 

Hunter-Liggett  Light  Red  Kill  Range 

t-Test:  Paired  Two  Sample  for  Means 


HLA  Warrior 

Janus 

Mean 

1142.378 

1044.613 

Variance 

418.9244178 

634.7711 

Observations 

10 

10 

Pearson  Correlation 

-0.07247826 

Hypothesized  Mean  Difference 

0 

df 

9 

tStat 

9.203287939 

p(T<=t)  one-tail 

3.5548E-06 

t  Critical  one-tail 

1.833113856 

P(T<=t)  two-tail 

7.10959E-06 

t  Critical  two-tail 

2.262158887 

Hunter-Liggett  Light  Rounds  Fired 

t-Test:  Paired  Two  Sample  for  Means 

HLA  Warrior 

Janus 

Mean 

15840.2 

71883.1 

Variance 

7943005.289 

18628348 

Observations 

10 

10 

Pearson  Correlation 

-0.12546122 

Hypothesized  Mean  Difference 

0 

df 

9 

tStat 

-32.5612979 

P(T<=t)  one-tail 

5.97732E-11 

t  Critical  one-tail 

1.833113856 

p(T<=t)  two-tail 

1.19546E-10 

t  Critical  two-tail 

2.262158887 
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